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FOREWORD 


Various  regulations,  such  as,  AFR  80- 5  and  69-29 »  and  AFSCR 
80-1  and  80-9  define  AF  policy  natters  in  Reliability  and  Maintain¬ 
ability  (R/M).  Equipment  and  management  specifications  such  as, 
MHj-R-275^2  and  MXL-M-2651 2  establish  program  or  equipment  require¬ 
ments,  but  the  AF  literature  generally  lacks  guidance  type  documents 
advising  the  SPO  or  procurement  activities  on  hov  to  establish  oar 
manage  an  R/M  Program. 

Availability  of  specialised  training  in  these  fields  is  limited, 
with  only  a  few  colleges  offering  courses.  Manpower  limitations 
further  complicate  the  training  programs  since  personnel  cannot 
be  spared  for  long  periods  of  time  to  take  those  courses  that  are 
available . 

The  purpose  of  this  TBR  is  to  provide  ready  reference  in  a  single 
volume,  information,  guidance5,  And  procedures  on  ESD  R/M  policy.  This 
volume  is  Organized  into  sections.  Each  deals  with  a  given  facet  of 
R/M  that  the  ESD  R/M  staff  has  noted  As  A  problem  area,  potential  or 
actual. 

This  document  is  not  a  study  in  depth,  nor  is  it  to  be  considered 
as  a  complete  and  final  text.  Rather  it  covers  present  ESD  philosophy 
and  provides  needed  guidance.  As  additional  work  is  done  in  the  R/M 
areas,  this  volume  will  be  revised  or  additional  sections  will  be  added. 
The  sections  contained  herein  were  prepared  during  the  1963-19^  period 
by  ESTE  staff  members;  G .  H.  Allen,  Major  J.  R.  Barton,  R.  M.  DeMilia, 
Capt  G.  Grippo,  and  J.  E.  Horowitz. 


V 


ABSTRACT 


The  ESD  Reliability  and  Maintainability  (R/m)  Staff  originaUy 
prepared  a  series  of  pamphlets  dealing  with  R/M  matters  during  1963-6^. 
These  have  now  been  combined  into  a  single  handbook  for  ready  reference 
and  assimilation  by  ESD  personnel  associated  with  R/M  programs.  Each 
section  of  this  handbook  deals  with  a  particular  problem  area  in  R/M 
matters  and  suggests  methods  of  initiating  and  operating  an  R/M  program. 
.1^^  The  material  covered  ranges  from  the  basic  elements  of  establishing  a 
program  thru  the  engineering  requirements  to  be  evaluated  in  design  re¬ 
views  .  The  overall  operations  involved  in  monitoring  of  a  contractors 
program  are  defined.  Several  sections  deal  with  the  mathematical  aspects 
_  ^  of  Reliability  decision  making  including  construction  of  probability  of 
acceptance  curves.  Specific  areas  covered  in  this  TER  are  listed  in  the 
Table  of  Contents. 


i 


REVIEW  AND  APPROVAL 


This  Technical  Documentary  Report  has  been  reviewed  and  is  approved . 


Chief,  Tech  Rqmts  &  Stds  Off 


ii 


V 


FOREWORD 
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80-1  and  80-9  define  AF  policy  matters  in  Reliability  and  Maintain¬ 
ability  (R/M).  Equipment  and  management  specifications  such  as, 
MlL-R-275^2  and  MIL- M- 26 512  establish  program  or  equipment  require¬ 
ments,  but  the  AF  literature  generally  lacks  guidance  type  documents 
advising  the  SPO  or  procurement  activities  on  how  to  establish  or 
manage  an  R/M  Program. 

Availability  of  specialized  training  in  these  fields  is  limited, 
with  only  a  few  colleges  offering  courses .  Manpower  limitations 
further  complicate  the  training  programs  since  personnel  cannot 
be  spared  for  long  periods  of  time  to  take  those  courses  that  are 
available. 

The  purpose  of  this  TDR  is  to  provide  ready  reference  in  a  single 
volume,  information,  guidance,  and  procedures  on  ESD  R/M  policy.  This 
volume  is  drganized  into  sections .  Each  deals  with  a  given  facet  of 
R/M  that  the  ESD  R/M  staff  has  noted  as  a  problem  area,  potential  or 
actual. 

This  document  is  not  a  study  in  depth,  nor  is  it  to  he  considered 
as  a  complete  and  final  text.  Rather  it  covers  present  ESD  philosophy 
and  provides  needed  guidance.  As  additional  work  is  done  in  the  R/M 
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The  sections  contained  herein  were  prepared  during  the  1963-196^  period 
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Capt  G.  Grippo,  and  J.  E.  Horowitz. 
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SECTION  I 

GUIDANCE  ON  PROPOSAL  CONTENT  FOR  RELIABILITY  AND  MAINTAINABILITY 
IN  SYSTEM/  EQUIPMENT  PROCUREMENTS 


FOREWORD 


Tne  purpose  of  this  section  is  to  provide  guidance  to  SPO  Reliability  and 
Maintainability  (r/m)  Monitors  in  establishing  requirements  for  tne  R/M  part  of 
a  system/equl pment  proposal. 

Specific  items  which  bidders  must  discuss  in  proposals  are  presented.  A 
brie:  discussion  is  presented  on  each  item.  This  section  amplifies  those 
instructions  contained  in  PMI  1-9,  Preparation  of  Requests  for  Proposals  for 
Systems,  2b  January  1963. 
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GUIDANCE  ON  PROPOSAL  CONTENT  FOR 
RELIABILITY  AND  MAINTAINABILITY  IN 
SYSTQVEQUUMENT  PROCUREMENTS 


1 .  Introduction: 

Current  approechee  to  the  disciplines  of  Relisbility  and  Maintainability  (R/M) 
include  the  insertion  of  quantitative  requirements  snd  R/M  specifications  in 
Request  For  Proposal  (RFP)  documents  which  deal  with  system/equipments. 

Responses  to  these  items  by  potent isl  contractors  hsve  been  varied  and,  at 
times,  have  left  important  R/M  considerations  unanswered. 

The  purpose  of  this  pamphlet  is  to  set  forth  the  R/M  items  which  s  potential 
contractor  is  expected  to  cover  in  e  technical  proposal.  In  addition,  significant 
materiel  to  be  presented  under  each  item  is  discussed. 

2.  R/M  Organization: 

A  bidder's  proposal  should  identify  the  position  of  hie  iy'M  Group  or  Section 
within  his  overall  organizational  structure.  Usually,  the  R/M  Group  either  is 
positioned  as  a  line  activity  under  engineering  or  combined  with  Quality  Control 
(and  perhaps  other  disciplines)  to  form  a  Product  Assurance  Department. 

Most  large  companies  will  have  a  R/M  Staff  Officer  (perhaps,  Vice-President, 
Reliability  and  Quality  Control)  who  is  responsible  for  generating  overall  R/fa 
policy  and  standard  operating  procedures. 

The  main  concern  in  studying  a  bidder's  R/M  organization  is  to  determine 
whether  or  not  the  proposed  organization  will  be  responsive  to  the  overall  R/M 
program  requirements,  sensitive  to  problem  areas,  and  sbla  to  contribute  to 
the  formulation  of  daeign  criteria  and  the  control  of  deaign  for  reliability 
and  maintainability. 

Its  ability  to  perform,  in  accordance  with  the  above  paragraph,  is  also  a 
function  of  its  personnel  capability-mix.  Since  R/M  encompeeaes  a  wide  variety 
of  tasks,  ranging  from  complex  modeling  techniques  to  design  criteria,  a  bidder 
oust  be  in  a  position  of  stating  the  quality  and  quantity  of  people  available 
to  perform  the  proposed  R/M  program. 

3.  Prediction  of  K/H: 

It  is  expected  that  a  bidder  will  make  a  first  determination  or  prediction 
of  the  R/fo  capability  of  hia  proposed  eye t em/equipment  design  and  compare  the 
results  with  the  quantitative  R/M  requirements .  All  equations  employed  in  tha 
computations  and  any  mat hematics 1  assumptions  must  be  clearly  stated.  In  addition, 
failure  rate  date  sources  and  a  diecuseion  of  their  adequacy  must  be  presented. 
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The  DT edict ion  serves  to  identify  possible  R/H  weakne.aeu  in  e  proposed 
svs tom/equl iment  design,  A  bidder  should  be  eble  to  indicete  bow  these  weaknesr.es 
to  to  o^coae.  For  example,  a  magnetron,  even  when  ^retet  oonservetively, 
has  e  high  (es  compared  to  s  simple  resistor,  for  example)  inherent  feilure 
™te  a£d  in  e  simple  reliebility  serieo  system/equiroent  influences  consi.erebly 
the  resultant  system/equipoent  failure  rete  A  biuocT  should  indice  e  wbet 
Mm«nsSing  features  (such  as,  redurdent  replecement  or  malfunction  protection) 
he  intends  to  introduce  into  bis  design  to  minimize  the  influence  oi  th.s  or 
other  high  failure  rate  items. 

Another  important  consideration  is  the  proposed  preventive  meirtenarce  cycle. 
Missions  can  bTinterrupted  by  unscheduled  (feilure.)  ss  well  es  scheouled 
maintenance .  The  problem  i.  the  uneveilebility  6f  ays tem/equi teen t  for  mission 
accomplishment  during  either  type  of  maintenance. 

The  manner  in  which  support  is  programmed,  efter  system/equtfmeni  transit .on , 
is  influenced  by  required  preventive  maintenance.  Bidders  should  be  expected 
to  dtocusa  the  amount  (frequency),  type, and  duration  of  preventive  maintenance 
required  for  their  proposed  system/equipment  designs. 

4.  ".vldence  of  Past  R/M  Accomplishments; 

o /y  Quantitative  requirements  have  been  inserted  into  Government  cor.t reels 
since  the  early  1950's.  It  is  to  be  expected  thet  bidders  will  be  able  to  c.te 
performance  on  past  contracts  end  stete  how  successful  they  were  in  mee  .np 

exceeding  imposed  quantitetive  requirements.  Since  programs  differ  ... 

Spee  ofso^htoUcation,  a  brief  description  of  overell  program  requirements 
should  be  included  with  each  program  cited. 

5.  R /M  Design  Review  Schedule; 

Potential  R/M  problems  must  be  detected  end  corrected  in  the  early  stages 
of  e  system/equipment  program.  Conservative  cost  estimates  indicate  .*e 
system/equipment  problems  which  remain  unresolved  during  design,  an  II  ^nelly 
ar^reaolveddurlng  operetionel  usege,  will  require  an  sxpendituro  of  one 
thousand  times  as  many  dollere  es  during  design  (to  sey  nothing  of  the 
inconvenienced*  having  system/equipmento  in  e  down-.t.te  while  moolfic.tr ons 
are  performed). 

The  following  types  of  design  reviews  are  expected  to  be  echeduled  by  e 
potential  contrector: 

e  Concept,  to  inveetlgete  and  decide  on  the  design  epproeches  to  be 
taken  to  aetlafy  the  quantitative  R/d  requirements . 

b  Component  Part  Selection  and  Application,  to  insure  thet  parts  having 
histories  of  “^failure  r.ta.  are  selected  to  be  Incorporated  into  sy.tem/equip- 
mentu  and  thet  conservative  epplicatlon  of  these  parts  takes  piece. 
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c.  Electrical,  to  insure  minimization  of  drift  type  failures,  simplicity 
of  design,  and  adequacy  of  failure  detection  devices. 

d.  Mechanical,  to  insure  proper  packaging,  design  of  the  cooling  system, 
nnd  overell  physical  layout  for  ease  of  maintenance. 

e.  Producibility,  to  alert  manufacturing  personnel  to  the  possibilities 
of  unique  or  unusual  manufacturing  techniques  that  may  be  required  during  the 
manufacturing  cycle. 

The  success  of  a  design  review  is  partially  dependent  on  the  participant*. 

Most  companies  will  essign  design  review  responsibilities  to  senior  or  staff 
engineers.  Usually,  there  will  be  a  permanent  design  review  chairman  who  will 
drew  on  the  technical  resources  of  a  company  es  needed. 

Minutes  of  meetings  are  expected  to  be  maintained  and  corrective  action 
recommendations  developed.  However,  a  decision  es  to  modification  of  e  design 
is  usually  left  to  the  appropriate  design  engineer,  unless  the  error  or  deficiency 
is  of  such  e  nature  es  to  warrant  referral  to  management. 

A  potential  contractor's  proposal  should  be  carefully  reviewed  to  determine 
the  adequacy  of  his  design  review  activity  in  terms  of  the  following  criteria: 

a.  Timeliness,  reviews  occur  before  drewing  release  to  production. 

b.  Frequency  and  variety  of  reviews. 

c.  Responsibility  for  corrective  action  follow-up. 

d.  Method  of  organizing  reviews  with  attention  to  the  type  of 
personnel  assigned  review  responsibility. 

6.  Description  of  Proposed  R/M  Program: 

A  bidder  is  expected  to  define  his  complete  proposed  R/M  Program  in  terms 
of: 


e.  Tasks  to  be  accomplished. 

b.  Task  descriptions. 

c.  Time-Phasing. 

d.  Significant  milestones. 

e.  Responsibility  for  task  accomplishments. 


Item  d  ia  important  since  it  serves  to  establish  program  monitoring  pointa 
or  time  a  at  which  contractor  progress  can  be  as sea sad  and  any  neceaaary  redirec¬ 
tion  given  by  the  procuring  agency.  Monitoring  of  contractor  programs  ia  a 
definite  procuring  agency  reaponaibility  under  current  Air  Force  Regulations 
(AFRa). 

RFPa  usually  contain  those  elements  of  an  R M  Program  which  are  compatible 
with  the  overall  ayatem/equipment  procuremant,  ESDP  80-2,  General  Requirements 
for  a  Reliability  and  Maintainability  Program  Plan  for  Electronic  Systems, 

15  August  1963,  haa  aet  forth  tha  beaic  R/fc  elements  which  are  applicable  to  all 
eya tam/a quipoent  procurements . 

Two  elements  or  taska  of  a  proposed  program  must  be  defined  in  some  depth, 
namely,  subcontractor  management  and  corrective  action  management. 

Sarious  ays tam/a quipment  R fii  problems  occur,  if  prima  contrectors  do  not 
provide  e  subcontractor  management  activity.  Thia  activity  should  ancompaas: 

a.  Incorporation  of  quantitetive  R/fo  requirements  into  all  specifica¬ 
tions  for  subcontracted  equipment. 

b.  Demonstration  requirements. 

c.  Provisions  for  a  R M  Program  which  is  compatible  with  the  Prime 
Contractor’s  Program. 

d.  Scheduled  monitoring. 

It  is  obvious  thet  the  collection,  processing,  and  analyzing  of  R/M  date  and 
the  holding  of  design  reviews  ae  such  cannot  improve  tha  R/M  capability  of 
aysteme/aquipments .  It  is  necaasary  to  supplement  thesa  tasks  with  a  corrective 
action  management  task.  A  bidder  should  be  prepared  to  outline  his  correct iva 
action  procedures.  These  procedures  should  include  e  discussion  of  hie  R/M 
deta  collection  feedbeck  system  (see,  for  example,  ESDP  80-3,  General  Require¬ 
ments  for  a  Dete  Collection  and  Evaluation  System  for  Electronic  Systems, 

1  November  1963)  end  specifically  should  indicate  the  provisions  by  which  his 
R/>{  Group  is  assured  the  opportunity  to  review  and  assess  the  effect  of  all 
changes  on  system/equipment  R fit  cepability. 

Since  design  or  inherent  reliability  must  be  protected  from  unnecessary 
degradation  during  manufecturing,  e  bidder  shou3d  discuss  his  factory  quaii  y 
control  system  with  particular  attention  to  any  unique  techniques  which  are 
to  be  employed  to  insure  delivery  of  reliable  equipment. 

These  techniques  could  involve  testing  samples  of  component  parts  to 
Acceptable  Reliability  Levels  (ARLs)  during  incoming  inspection,  additional 
"burn-in'*  teats  of  major  elements  of  equipment,  specialized  training  courses  o 
increase  skill  levels  of  personnel  involved  in  manufacturing,  etc. 
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7.  Discussion  of  Equipment  R/fo  Demonstration  Plans; 

Tha  specification  of  quantitative  R/fa  requirements  for  equipment  involvea 
consideration  of  stetisticel  techniques  to  illustrate  that  the  requiramenta  in 
feet  heve  bean  satisfied. 

ESDP  80-5,  Varificetion  of  Quantitetive  Reliebility  Requirements  (Decision 
Criterie),  15  November  1963,  sets  forth  some  approved  reliebility  demonstration 
techniques.  An  important  conelderation  etreased  in  ESDP  80-5  is  tha  quantifica¬ 
tion  of  tha  risks  involved  in  making  decisions  ebout  compliance  to  reliehillty 
requirements . 

An  ecceptabla  model  for  decision-making  purposes,  undar  certain  circuaetancae, 
ia  tha  Cummulativa  Poisson  Distribution.  Thia  model  has  as  beaic  inputs: 

a.  A  fixed  value  for  teat  time  (T). 

b.  A  velue  for  maan-time-between-feilure  (MTBF,  0). 

c.  An  ellovebla  number  of  failures  (C). 

A  choice  of  C  should  be  made  aTtar  a  study  of  operating  charecteristic 
functions.  Such  functions  relate  tha  probability  of  acceptance  to  values  of 
the  retio  of  "true”  MTBF  (9j)  to  contractual  MTBF  (0).  For  any  C,  es  this 
ratio  incraeaea  in  valus,  the  probability  of  ecceptance  increeaes. 

For  axample,  assume  e  test  time  aqual  to  e  contractual  MTBF.  If  C  is  sat 
equal  to  zero,  labia  I  indicates  that  an 

Tahie  1 

c-o 

Probability  of  Acceptance  (Approx.) 

2% 

u$ 

37$ 

61$ 

4  78$ 

6  84$ 

10  90$ 

equipment  would  have  to  be  delivered  for  test  with  e  "true"  MTBF  ten  times  the 
value  of  the  contrectuel  MTBF  in  order  to  heve  a  90$  probehility  of  acceptance. 
Such  e  situation  probebly  would  reeult  in  e  bidder  requesting  thet  C  be  raised 
to  another  velue.  If  C  were  set  equal  to  one,  for  example,  the  probability  of 
ecceptance  values  would  be  es  irriiceted  in  Teble  2. 


°l/0 

V4 

1/2 

1 

2 
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Table  2 
C=1 


V® 

1/4 

1/2 

1 

2 


Probability  of  Acceptance  (Approx) 


1 0* 
42* 
74* 
91* 


Table  2  number e  Indicate  that  an  equipment  would  hare  to  be  delivered  for  test 
with  e  "true"  MTBP  two  times  the  value  of  the  contractual  MTBF  in  order  to  have 
et  least  e  90*  probability  of  ecceptanoe. 

The  arithmetic  also  determines  the  procuring  activity  risk  or  probability 
of  accepting  an  unsatisfactory  equipment.  For  example,  Table  2  numbers  indicate 
that  there  is  e  10*  probability  that  an  equipment  with  e  true  MTBF  of  oniy 

the  required  will  be  sccepted;  i.e.,  experience  one  or  l.c  f.il^e. 

during  the  test.  On  the  other  hand,  e  bidder  1 Ce  e^i 

e  "true"  MTBF  twice  the  value  of  the  contrectual  MTBF  and  atill  have  a  9* 
probability  that  the  equipment  would  be  rejected;  i.e.,  experience  more 
one  failure  during  the  test. 

Additional  computations  could  be  performed  by  setting  test  time  equal  to 
multiples  of  contrectual  MTBF.  The  point  of  this  discussion  is  to  alert 
proposal  evaluators  to  the  problem  of  selecting  e  C  value  with  due  regard  to 
the  risks  involved. 

Maintainability  demonstration  involves  simulating  equipment  feiluree' to 
daveloo  e  statistically  significant  repair  time  sample  sice.  While  MIL-M-26512 
outlines  e  method  for  maintainability  demonstration,  other  methods  proposed  by 
bidders  should  be  examined  for  their  ecceptebility.  The  Important  consideration 
is  thst  en  approved  quantitative  decision  rule  be  developed  Oi2£  to  the 
commencement  of  demonstration. 

8.  Summary: 

A  bidder  is  expected  to  consider  the  following  R/H  criteria  in  preparing 
his  sys tea/e quipment  proposal: 

e  Prediction  of  the  R/M  capability  of  his  proposed  system/equipoent 
design  with  attention  to  potential  R/H  problem  areas  and  approaches  to  problem 
resolution. 

b  An  in-depth  discussion  of  design  review  activities,  corrective 
action  management  plans,  and  nfit  demonstrstion  techniques. 

c  A  task  by  task  description  of  his  proposed  R/V.  Iropran.  Inis 
description  should  include  a  time-phasing  of  appropriate  milestone  review 
points . 
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d.  An  identification  of  his  R/fo  Organization,  its  capability-mix, 
lines  of  communcietion  and  responsibilities. 

e.  A  description  of  eccompliahments  on  past  r/M  Programs,  compliance 
to  quantitative  R/H  requirements ,  and  an  indication  of  the  sophistication  of 
cited  overall  system/equipment  procurements. 
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SECTION  1  I 


GENERAL  REQUIREMENTS  FOR  A 
RELIABILITY  AND  MAINTAINABILITY 
PROGRAM  PLAN 
FOR 

ELECTRONIC  SYSTEMS 
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SECTION  II 

GENERAL  REQUIREMENTS  FOR  A  RELIABILITY  AND  MAINTAINABILITY 
PROGRAM  PLAN  FOR  ELECTRONIC  SYSTEMS 


FOREWORD 


Tne  purpose  of  this  section  is  to  provide  guidance  to  SPO  Reliability  and 
Maintainability  (R/M)  Monitors  in  establishing  general  requirements  for  a  con- 
tractor's  "R/M  Program  Plan*’. 

Underlying  this  section  is  a  fundamental  principle:  ESD  will  not  state 
to  a  contractor  that  an  R/M  program  is  to  be  performed 

existing  R/M  specifications.  Existing  specifications  are  by  necessity  q  1 
genera 1  In  nature.  Their  requirements  must  be  selected  and  defined  on  t  e 
fcasis  of  individual  system/equipment  procurements. 

However,  there  is  a  set  of  fundamental  R/M  tasxs  that  are 
mandatory  for  any  system/equipment  procurement.  The  exact  depth  of,  and 
approach  to“  these  iasks  will  of  course,  be  dependent  on  individual  system/ 
equipment  procurements. 

It  is  recommended  that  the  SPO  provide  the  Using  Command  , and 

3n^E*2?.s  sraSiTiaa:  rrta.ssJ- 

accepting  a  system  due  to  unresolved  defimtiong  of  R/M  acceptability. 

Assistance  in  defining  R/M  program  requirements  is  given  to  SPOs  by  the 
Office  of  Primary  Responsibility  (OPR)  in  accordance  with  ESDR  80-2  and 
ESDR  80-4. 


ii 


SECTION  II 


CONTENTS 

SUBJECT 

PAGE 

1.  Basis  of  Requirement  for  a  Reliability  and 

Maintainability  Program  Plan  . 

2.  Format  of  Program  Plans  .  ^ 

3-  Fundamental  Reliability  and  Maintainability 

Program  Elements  or  Tasks  .  2 

Actions  Required  by  ESD  .  q 

APPENDIX 

1.  Some  R/M  Program  Interfaces . . .  ^ 

ILLUSTRATION 

FIGURE 

I  Systems,  Reliability  and  Maintainability 

Engineering  Coordination  .  ^ 


n; 


GEKERAL  RTQUIEOffiJrrS  FOR  A  RELIABILITY  AXD 
HAINTAIK ABILITY  PROGRAM  PLAH  FOR  KLECTROSIC  SYSTD4S 


1.  Bwaia  of  Requirement  for  j  Reliability  antj  Maintainability  Program  Plant 

'  MIL-R-27542,  paragraph  3.3,  and  MIL-M-26512,  paragraph  3.2,  raquiraa  tha 
preparation  by  a  contractor  and  submission  to  tha  procuring  agency  of  a 
reliability  and  a  maintainability  plan,  respectively. 

The  Electronic  Systems  Division  (ESD)  discourages  the  submission  of  two 
plans;  one  for  Maintainability  and  one  for  reliability.  This  is  tha  result  of 
the  unique  o'paratlonal  requirements  of  systems  assigned  to  ESD.  These  require- 
manta  involve  a  numerical  expression  for  operational  availability.  The  latter 
represents  a  simultaneous  treatment  of  maintainability,  expressed  as  a  mean- 
down-time  (MDT)  statistic;  and  reliability,  expressed  as  a  mean-time -be tween- 
failure  (KTBF)  statistic. 

ESD  procures  one  combined  program  plan  which  most  describe  the  methods  by 
which  e  contractor  controls  both  the  MDT  and  MTBF  characteristics  of  electronic 
systems  beginning  with  the  design  phase  of  a  system  program.  This  plan  can  be 
referred  to  as  an  "Availability  Program  Plan." 

2.  Egmi  sL  ftrcgrw  JQm» 

First,  ESD  expects  a  program  plan  to  provide  * 

a.  The  tasks  or  work  elements  to  be  accomplished. 

b.  A  description  of  the  work  to  be  accomplished  under  each  task  (task 
description). 


c.  The  time-phasing  of  each  task. 

d.  The  manloeding  assigned  for  the  accomplishment  of  each  task. 

e.  Appropriate  program  plan  milestone  review  points. 

Secood,  as  exhibits  or  attachments  to  a  plan,  ESD  expects  to  have  a 
contractor  describe  his: 

a.  Design  review  system,  its  method  of  operation,  responsibilities, 
and  authority. 


b.  Corrective  ection  system,  Including  his  data  collection  system,  and 
proposad  computational  ground  rul es. 

c.  Changa  order  control  system,  with  particular  attention  to  the  Method 
by  which  hla  R/M  organisation  has  the  opportunity  to  review  all  design  changes 
for  quantitative  effects. 

Third,  as  exhibits  or  attachmanta  to  a  plan,  ESD  expects  to  have  a  con¬ 
tractor  indicate  the  position  of  hla  R/H  operation  within  his  management  struc¬ 
ture,  describe  the  organisation  of  this  operation,  and  indicate  the  channels  of 
communication  between  this  organisation  and  design  engineering,  quality  oontrol, 
test  engineering,  and  components  engineering. 

Fourth,  ESD  discourages  the  preparation  of  pro  gras  plan  reports  on  offset 
printing  with  elaborate  covers.  Hecto  or  engineering  letter  type  reports  are 
preferred  for  purposes  of  coat  reduction. 

3.  gapdflflinlii  Bfllflfrlltti  tad  HalirttlMMlltr. Elamite  gr  ImRr* 

a.  The  family  of  reliability  specifications  (MIL-R-27542.  KIL-R-27070, 

MIL-R -26474)  end  the  maintainability  specification,  MIL-M-26512,  actually 
describe  a  serlas  of  reliability  and  maintainability  tasks  or  work  elements. 
Depending  on  the  type  of  procurement,  the  nature  of  the  quantitative  require¬ 
ments,  and  the  importance  of  the  system/equipment  mission,  ESD  will  deolde  on 
the  reliability  and  maintainability  tasks  to  be  aooompllshmd  by  e  oontrector. 
However,  there  are  certain  R/fc  tasks  which  are  considered  fundamental  to  the 
establishment  of  an  acceptable  R/fc  program.  These  tasks  can  be  grouped  for 
convenience  under  the  following  general  headings  t 

(1)  Ha  thamati  cal /Statistical  Analysis. 

(2)  Design  Assistance. 

(3)  Assessment  and  Verification. 

(4)  Test  Planning  Assistance. 

(5)  Subcon treotor  Management  (For  Prime  Contractors). 

(6)  Failure  Analysis. 

(7)  Corrective  Action  Management. 

(8)  Manufacturing  and  Field  Support. 

(9)  Reports. 
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b.  Task  Descriptions: 

(1)  Mathematical /Statistical  Analysis.  ESD  requires  the  preparation 
of  e  mathematical  model  which  allows  the  computation  of  the  appropriate  R/H 
statistics  describing  the  system  under  development.  A  first  step  in  the 
construction  of  such  e  system  model  is  the  performance  of  an  "equipment  block 
analysis."  This  analysis  will  indicate  the  possible  modes  of -operation  or  con¬ 
figurations  which  allow  mission  accomplishment.  For  equipment  in  reliability 
aeries,  the  block  analysis  results  are  obvious.  However,  for  the  more  complex 
equipment  configurations  under  development  at  ESD,  the  analysis  Is  usually  not 
routine.  Following  the  block  analysis,  appropriate  mathematical  expressions 
will  be  written. 

It  is  expected  that  the  quantitative  requirements  will  be  appor¬ 
tioned  over  equipments,  subassemblies,  and  critical  or  high  failure  rate 
piece  parts .  This  apportionment  actively  serves  as  a  design  control.  Design 
engineers  are  enxioua  to  know  the  burden  placed  on  their  particular  design  by 
the  overall  quantitative  requirements . 

Several  R/fa  predictions  are  expected  to  be  produced  during  a  pro¬ 
gram.  The  exact  time-phasing  of  these  predictions  is  e  function  of  the  overall 
program  schedule.  ESD  will  not  be  satisfied  by  Just  the  reporting  of  a  result. 
All  computations  will  be  supported  by  citing  appropriate  failure  rate  sources 
(such  es,  MIL-HDEK-217,  "Relleblllty  Stress  and  Failure  Rate  Data  for  Elec¬ 
tronic  Equipment").  If  unique  or  contractor  oriented  failure  rates  are  employed, 
ESD  expects  to  have  Justification  presented  for  their  use.  This  Justification 
will  lncluda: 


(a)  Method  of  data  collection  and  reduction. 

(b)  System /equipment  and  operations  over  which  the  data  was 
collected  and  analyzed. 

(c)  An  adequate  explanation  of  the  use  of  any  extrapolations 
or  adjustment  factors. 

Statistical  analysis  will  be  performed  in  support  of  all  test 
activities.  This  analysis  serves  to  express  quantitatively  tha  R/fr  charac¬ 
teristics  of  system/equipment.  In  addition  to  such  statistics  as  MTB7,  MDT, 
and  Availability,  ESD  recognizes  the  usefulness  of  the  development  of  e  relia¬ 
bility  function  -  the  statistical  probability  of  no  failures  es  a  function  of 
satisfactory  operating  time,  and  the  maintainability  function  -  the  statls tidal 
probability  that  a  system/equipment  will  be  restored  to  a  satisfactory  opera¬ 
ting  condition  es  e  funotion  of  down  time.  Initially,  such  functions  may  be 
constructed  by  application  of  non-parametrlc  statistical  teohniquss  and 
eventually  fitted  to  underlying  probability  density  functions. 
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lie  fore  stp-tisticel  computations  take  place,  it  is  necessary  thst 
ESD  and  a  contractor  agree  to  computational  "ground  rules."  Quoting  of  numbers 
without  a  thorough  understanding  of  the  techniques  employed  in  their  develop¬ 
ment  is  unacceptable  to  ESD. 

A  further  important  use  of  statistical  analysis  is  the  identifica¬ 
tion  of  "week-links"  in  syatem/equipment  and  a  support  to  the  corrective  action 
process . 

(2)  Design  Assistance.  A  contractor’s  R/M  organization  is  expected 
to  supply  design  engineering  with  recommendations  and  techniques  for  designing 
reliability  end  maintainability  into  a  system/equipment.  As  examples,  techniques 
for  designing  reliability  into  a  system/ equipment  are: 

(a)  Conservative  application  of  component  parts  (appropriate 
margins  of  safety). 

(b)  Circuit  simplification. 

(c)  Redundant  replacements  and/or  alternate  modes. 

(d)  Minimization  of  environments 1/oper at ional  stress. 

Formal  engineering  design  reviews  are  required  to  be  scheduled 
at  significant  points  in  a  program.  These  reviews  will  range  from  parts  list, 
part  applications,  fail-safe  circuit  practices,  sirplif icetlon  of  circuitry, 
use  of  s^erderc  circuits,  mechanical  and  packaging  considerations,  to 
resolution  of  interface  R/M  problems  of  equipments.  ESD  expects  to  participate 
in  selected  reviews  and  a  contractor  must  alert  ESD  ten  days  prior  to  conduct¬ 
or,-  any  review.  Ir  addition,  a  contractor  must  raintain  complete  records  of 
each  review  end  furnish  them  to  ESD  upon  request.  A  summary  of  the  results 
of  each  review  will  be  made  ac  part  of  monthly  progress  reports. 

All  r.on-SCFs  end  BCPs  will  be  reviewed  for  Quantitative  effects 
or  reliability  and/or  jr.nirt airabili ty. 

Continuous  liaison  between  »  cor  tractor's  R  fi\  end  design 
organization  will  be  maintained  to  assure  that  timely  corrective  action  is 
taker  or  R/M  "weak -links . " 

(3)  Assessment  snd  Verification.  It  is  absolutely  essential  that  a 
contractor ,c  R/M  Program  provide  for  e^d'emonstretion  tbsi  the  contractual 
quen+ita<ive  reliability  end  maintainability  requirements  have  been  achieved 
and /or  exceeded.  The  means  by  which  this  verification  is  to  be  accomplished 

v  11  oe  se*  forth  by  ESD.  Usually,  due  to  the  complexity  of  electronic  systems, 
r  combination  of  enalytical  and  test  methods  is  employed. 


The  contractor* a  K/H  plan  will  provide  for  the  collection  of 
failure,  downtime,  and  repair  time  deta.  The  data  collection  format  will  be 
subject  to  the  approval  of  ESD.  Statistical  reduction  of  collected  data  haa 
bean  diacuaaed  in  paragraph  3b(1)  above. 

(4)  Tact  Planning  Aaaistance.  A  contractor' a  R/fc  organization  will 
participate  in  the  development  of  fiH  teat  plana,  eepecially  those  involved 
with  category  tea ting,  for  system* /equipments ♦  Particular  emphasis  ia 
naceaaary  ons 

(a)  The  atatiatical  and  engineering  validity  of  plana,  e.g., 

tba  propertiea  of  randomization  and  replication,  are  aaeential  to  a  valid  plan. 

(b)  Tba  timelinaae  and  accuracy  of  the  failure  deta  collection 

eye  tern. 

(5)  Subcontractor  Management.  Prime  contractors  will  be  required  to: 

(a)  Incorporate  quantitative  R/fc  requirements  in  subcontracted 
equipment  specifications. 

(b)  Assure  that  each  subcontractor  has  an  R/fo  program  which  la 
compatible  with  the  overall  R/fa  program.  Subcontractor  progress  will  be 
periodically  reviewed. 

(c)  Attend  and  participate  in  subcontractor  engineering  dap'gn 

reviews .  *  * 

(d)  Review  subcontractor  predictions  amd  computations  for 
accuracy  and  correctnaaa  of  approach. 

(a)  Furnish  subcontractor*  with  failure  ani  maintainability  data 

resulting  from  teata.  7 

(f)  Require  subcontractor  progress  reports. 

(g)  Review  subcontractor  teat  plana  for  accuracy  and  correctness 

of  ftpprotcb « 

(b)  Assure  that  subcontractor*  hava,  and  are  pursuing,  a 
unreli*bilitr#Ct^T"  *ction  a^fort  011  causes  of  unmaintainability  end /or 

ESD  ia  aware  that  causes  of  unmaintainability  and/or  unreliability 
can  arise  from  poor  communications  and  monitoring  of  subcontractor*  by  crime 
contractors.  ESD  will  periodically  visit  subcontractors  to  determine  the 
effectivenese  of  prime -subcontractor  R/H  program. 
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(6)  AiWWmla.  Maintenance  of  record*  or  statistical  *o*ly*l* 

of  data  i*  not  ocmpletsly  sufficient  to  support  a  oorractlta  *ctlom  * 

ijfiii.  Baoord  analysis  mo*t  to  *uppl— «tmd  to  anglnwlna  laboratory  analytic 
of  calactad  fail*d  oompooent  parte,  unite,  and/or  acaaahliac.  I?  ftTTt  A 
neefnl  ouroosa.  the  failure  a  naira  la  effort  must  to  _MIf  fttUl  la^glW  IfiW 
the  corrective  action  CTltf « 

17)  c^tiw  Action  miMWMnt.  d*t*.“^*i*’  *"* 

failure  analytic  era  not  completely  adequate  to  tofovve  thm  reliability  or 
■ainta inability  of  a  cycte^quijmant.  Theca  actloma  net  be  supplemented  hi 
a  corrective,  action  cycle*  that i 

(a)  Assign*  recponcibilitiec  for  corrective  action. 

(b)  AiciffC  cucpance  dates  for  completion  of  the  required  action. 

(c)  Provide*  f cllow-up  to  accure  that  action*  are  actually 

taken. 

(d)  Acceacec  the  qimntitetiv*  affect  on  reliability  and/or 
maintainability  by  the  action. 

(a)  Accure*  that  deciffi  principle*  are  followed  in  any 
proposed  modification. 

(f)  Determinac  the  affactivity  of  a  •fix*. 

(c)  Maintain*  c  problem  or  -veck-link-  lict  by  contractor*. 

Thlc  lict  dll  oontaini 

1.  Definition  or  etatement  of  a  problem. 

g.  Corrective  action  contemplated. 

2.  Action  agency  or  res  possibility  for  problem  reeolutio*. 

4.  Effect  cf  problem  on  reliability  and/or  maintainability. 

2.  Action  completion  date. 

An  effective  contractor  aamfment  tool  i*  an  Tm|rovem»t 
CoMittea.  T bis  oc-dttee,  oompoead  cf  aambarc  from  organiaationc  involved 
in  ©btainin*  corrective  action,  e.g.,  decign  angin^ring,  ompoomti  enginmmring, 
mmlito  oontrel,  purobaelng,  reliability,  etc.,  meat*  periodically  to  assure 
thatprogreas  ia  toingaade  to  raeolce  "weak-linka*  and  to  asalgD  respoosi- 
bdiity  on  aMitiomal  problems. 
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ESD  recognises  tha  complexity  cf  K/H  problems  and  the  necessary  inter¬ 
actions  between  organisational  groups  in  ordar  to  obtain  timely  corrective 
actions .  Such  a  committee,  therefore,  la  recognised  as  a  useful  element  of  a 
contractor's  corrective  action  syetem. 

(8)  Manufacturing, and,  field  Support.  Tha  K/K  designed  into  equipments 
must  not  be  allowed  to  be  significantly  degraded  during  the  manufacturing  and 
site  installation  phases  of  a  ayetem/equlpment  program  by  the  introduction  cf 
”Q.C.  type  of  failures*,  furthermore,  the  downtime  of  a  system/eqmlpment  most 
□ct  be  allowed  to  increase  because  of  improper  provisions  for  replacement* 
(spares)  and/or  the  unreliability  of  AGE. 

Technical  manuals  have  also  been  a  source  of  unreliability  and/or 
unmainta inability.  Shipping  and  storage  practices  have  produced  additional 
failures  into  a  ayatem/equipment. 

Improper  positioning  cf  equipment  within  a  shelter  has  Introduced 
"buman-cauaed  failures"  and  contributed  to  increased  system/equipment 
downtime.  Therefore,  during  manufacturing  and  Installation,  ESD  will  require 
a  contractor  to  conduct  an  "R/k  assurance  support  effort*. 

In  addition,  to  assure  the  timely  and  accurate  transmission  of  failure 
and  maintainability  data  from  testa  on  site,  the  contractor  will  assign  B./H 
engineers  to  a  site  who  are  charged  with  tha  responsibility  cf  data  collection. 
These  engineers  will  also  make  a  preliminary  classification  of  each  failure  ec 
to  causa,  a.g.,  design  error,  component  part,  mishandling,  operational  error, 
manufacturing  or  fabrication,  ate.,  and  affect,  a.g.,  lethal,  Mtjor,  minor, 
and  no  effect.  This  claeaifi cation  will  be  reviewed  by  the  main  R/fo  organisa¬ 
tion  ae  part  of  the  determination  of  "weak -links"  and  assignment  cf  corrective 
action  responsibility. 

The  on-site  R/fr  engineer*  will  be  kept  informed  of  tha  program*  made 
in  taking  corrective  action*. 

(9)  Reports  t 

(a)  ESD  expects  a  contractor  to  summarise  monthly  his  progress 
on  each  R/fr  task. 

(b)  Special  reports  will  be  required  on  mathematical  modal 
development*,  predictions,  demonstration  plan*  and  result*,  and  design  review 
result*. 


(c)  ESD  will  request  minutes  of  design  review  meetings,  relia¬ 
bility  indoctrination  lectures,  and  failure  analysis  summaries. 
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(d)  Vbenev*r  any  of  tho  abora  report*  contain  aignlf leant 
scientific  or  tacbnlcal  data  tba  report  will  ba  prepare!  and  published  aa  an 
BSD  Tacbnlcal  Documentary  Baport  In  aocordanoa  with  Vol«a  III,  BSD 
Contractor  Eaporta  Exhibit  63*1 • 

ESD  will  require  a  oontractor  to  indicate,  a a  part  of  hla  program  plan, 
tubmlaslon  dataa  for  tba  above  material, 


4,  let  Iona  Required  br  ESPi 

a.  General  R/H  atatamanta  aai  "Reliability  will  ba  in  aocordanoa  with 
MIL -R- 27 542"  or  Bone  other  reliability  ■  pacification,  or  "Maintainability 
will  ba  In  accordance  with  MIL-M-26512,"  praaant  unglUt! flfitad  guidance  to 
potential  contractor*.  It  la  pace jparr  that  an  B/tt  oreiraq  frt 
ault  the  need  a  frpW»<niUaS£- 

the  general  design  of  an  R/H  program  are  aat  forth  in  ESDR  80-2  and  ESDR  80-4. 


b.  ESD  will  aat  the  date  for  eutndttal  of  a  contractor's  proposed  R/ta 
program  plan.  A  proposed  plan  will  ba  thoroughly  reviewed  for  it*  acceptability 
against  the  program  design  established  by  ESD. 


c.  Comments  will  be  presented  on  the  proposed  plan  either  verbally, 
during  a  contractor  guidance  meeting,  or  in  writing  (or  both).  A  re  submittal 
date  for  the  plan  will  be  established. 

d.  A  contraotor  will  receive  formal  notification  of  aooaptanoe  of  the  plan. 


a.  ESD  will  not  give  approval  of  a  prime  contractor’s  plan  until  each  sub¬ 
contractor's  plan  has  been  reviewed. 


f.  ESD  will  require  that  a  daflnit#  schedule  of  program  reviews  be 
established.  These  reviews  are  in  addition  to  attendance  at  contractor 
engineering  design  reviews. 


APPENDIX  I 


Some  R/M  Program  Interfaces 

A  contractor's  R/M  Program  must  be  carefully  integrated  within 
his  total  effort,  since  the  resulting  reliability  and  maintainability 
char ac ter 1 sties  of  a  system  are  influenced  by  system  design,  hardware 
or  equipment  design,  test  equipment,  number  and  skill  level  of  per¬ 
sonnel,  training,  technical  manuals  and  physical  or  operational 
environment.  But,  it  is  during  tbe  early  system  engineering  phases 
of  a  program  that  an  R/M  Organization  can  contribute  significantly 
to  achievement  of  system  requirements . 

For  example,  ESD  might  state  a  system  point  availability  require¬ 
ment  which  is  expressed  as, 

A  .  MTBF _ 

MTBF+  MTTR 

To  establish  specific  subsystem  and  major  equipment  R/M  require-, 
merts,  the  system  point  availability  requirement  must  be  analyzed 
within  a  framework  which  includes : 

a.  Functional  or  performance  requirements  wbicb  dictate  general 
design  features . 

b .  Cost  constraints  which  can  be  allocated  between  initial  coats 
(equipment  research  and  development,  production,  installation,  train- 

technical  manuals,  special  equipment  and  tools),  and  support  costs 
which  continue*  throughout  the  life  of  the  system. 

c.  Time  constraints  including  design,  production,  installation  and 
training. 

d.  Personnel  constraints  which  dictate  the  general  skill  level  and 
number  of  personnel  available  for  operating,  controlling  and. maintain¬ 
ing  the  system. 

e.  Miscellaneous  requirements  and  constraints,  such  as  established 
support  and  logistic  policies,  environmental  conditions,  etc. 

Witb  considerations  such  as  expressed  in  (a  -  e),  it  is  obvious  that 
allocation  of  a  system  point  availability  requirement  to  subsystems  and 
major  equipments  must  be  cooperatively  performed.  Figure  I  suggests  a 
simple  flow  by  wbicb  reliability  and  maintainability  considerations  are 
included  wltbin  the  total  system  engineering  decisions. 


A  number  of  hardware  approaches  have  been  derived  as  possible 
alternative  methods  for  restoring  a  system  to  operation  subsequent  to 
a  failure  of  a  subsystem.  Three  major  requirements  must  be  considered 
within  each  method: 

a.  Detection  time  for  the  presence  of  a  failure 

b.  Localization  time  for  a  failure 

c .  Restoration  time  for  the  system  to  achieve  satisfactory 
performance 

These  requirements  interface  with  methods  for  designing  for  reli¬ 
ability.  Thus,  the  need  for  coordination  between  reliability,  maintain¬ 
ability  and  system  engineers. 

With  overall  system  responsibility,  ESD,  by  scheduled  monitoring 
visits  and  attendance  at  design  reviews, can  help  in  fostering  ccramuni- 
cation  between  all  organizations  influencing  system  design  and  develop¬ 
ment. 
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SECTION  I  I  I 

GUIDANCE  FOR  RELIABILITY  AND  MAINTAINABILITY 
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SECTION  III 

GUIDANCE  FOR  RELIABILITY  AND  MAINTAINABILITY  ENGINEERS 
PARTICIPATING  IN  CONTRACTOR  DESIGN  REVIEW  PROCESS 


FOREWORD 


1  Puroose.  The  purposes  of  this  section  are  to  provide  information 
and  gui5an«  on  reliability  and  maintainability  engineering  design  reviews 
required  by  MIL-R-27&42  and  MIL-M-26512,  and  to  outline  approaches  usually 
taken  by  contractors  in  accomplishing  such  reviews. 

o  Tw  wh™  Tt.  Applies.  The  contents  of  this  section  apply  to  all  programs 

falling  within  the  p^ie»  of  ESD  that  will  result  in  equipment  ("hardware 
£ distinguished  from  "software")  entering  the  Air  Force  inventory  and  ^ere 
the  engineering  design  review  requirements  of  MIL-R-27542  and  MIL-M  265 
are  a  specific  contract  requirement.  ESD  personnel  will  P^t^cipa 
reviews  to  assure  proper  design  consideration  of  equipment  reliabil  y 
and  maintainability. 
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Chapter  1 
IBTRODOCTKW 


A  design  review  Is  defined  as  a  planned  continuous  monitoring  of  a 
product  design  to  assure  that  It  meets  the  expressed  and  Implied  perform¬ 
ance  requirements  of  the  equipment  during  operational  use.  Such  a  review 
provides  periodic  appraisal  of  the  design  effort  to  determine  the  progress 
being  made  in  achieving  the  design  objectives  and  systematically  brings 
to  bear  specialized  talent  on  specific  problem  areas.  In  this  matter  an 
overall  evaluation  Is  made  to  take  into  consideration  specific  design  and 
Interface  problems  that  may  be  encountered  later  In  the  development  and 
production  cycle. 

Reliability,  maintainability,  and  value  ao  well  as  other  characteristics 
are  affected  by  every  decision  the  design  engineer  makeu.  This  includes 
the  choice  and  use  of  circuits  and  component  parts,  their  arrangement,  the 
environment  in  which  the  equipment  will  be  used,  to  the  capabilities  and 
problems  of  the  man  who  ultimately  must  operate  the  equipment  under  field 
conditions  that  could  not  have  possibly  been  foreseen  during  design. 

Although  the  design  engineer  is  not  solely  responsible  for  the  operational 
reliability  and  maintainability  of  equipment,  it  is  very  easy  to  blame  him 
when  equipment  has  a  poor  reliability  and  maintainability  field  record. 

It  should  be  emphasized  that  reliability  and  maintainability  are  not  the 
sole  responsibility  of  the  designer,  ncr  is  designing  a  responsibility  of 
the  design  review  team  and  every  effort  should  be  expended  to  avoid  such 
tendencies.  Because  of  Increased  complexity  cf  the  equipment  being 
designed,  concurrency  concepts,  and  new  devices  and  techniques  being 
employed,  it  Is  impossible  for  a  design  engineer  to  maintain  excellence 
in  every  technical  discipline  affecting  the  design  process.  To  obtain 
maturity  of  design  necessitates  evaluation  by  technical  specialists 
selected  for  their  special  talents  and  knowledge  who  perform  a  technical 
analysis  of  the  system/equipment  as  It  pertains  to  their  specialized  fields 
The  design  review,  if  properly  performed,  provides  one  of  the  most  powerful 
and  effective  tools  available  to  assure  that  reliability,  maintainability, 
and  value  as  well  as  ether  design  characteristics  have  been  considered 
early  In  design  or  at  the  optimum  stage  of  development. 

A  purpose  of  an  engineering  design  review  Is  to  analyze  system/equip¬ 
ment  requirements,  electromechanical  and  electronic  circuit  design,  component 
part  applications  and  mechanical  features,  to  assure  essential  characteristics 
such  as  reliability  and  maintainability  at  the  lowest  overall  cost. 

To  achieve  these  ends,  system  program  personnel  will  enforce  and  actively 
participate  In  a  design  review  process  on  all  programs  that  result  in 
equipment  entering  the  Air  7orce  Inventory. 

Techniques  for  designing  reliability  into  system/equipments  usually 
are  classified  into  three  categories: 
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a.  Conservative  selection  and  application  of  piece  parts.  (See 
RADC  Reliability  Notebook) 

b.  Incorporation  of  redundant  replacements  and/or  alternate  modes 
of  operation. 

c.  Minimization  of  environmental  stresses;  for  example,  electronic 

equipment  must  incorporate  means  for  adequate  heat  rejection  in  order  to 
provide  reliable  performance  at  thermal  e  .'.ilibr  ium .  It  canno  . , 

emphasized  that  reliability  can  be  obtained  only  if  the  electronic,  therma  , 
and  mechanical  designs  are  veil  executed.  The  thermal  design  is  fully 

iTnrwir+ant  as  the  circuit  design.  Ground-based  electronic  equipment  is 
frequently  installed  in  shelters  having  a  ventilating  or  air  conditioning 
system  intended  for  the  comiort  of  operating  personnel.  The  cooling  system 
for  the  electronic  equipment  must  be  made  compatible  with  sue  a  sys  e 

A  number  of  maintainability  design  techniques  have  been  ^lved  as 
possible  alternative  methods  for  restoring  a  system/equipment  to  operatl 
subsequent  to  a  failure .  Three  major  requirements  present  in  system/equip- 
merits  are : 

a.  Methods  for  detection  of  the  presence  of  a  failure. 

b.  Methods  for  localizing  a  failure  to  a  replaceable  unit  or  assembly. 

c.  Methods  for  restoring  operation  after  localizing  the  failure. 

Since  the  majority  of  system/e quipments  under  development  by  ESD  have 
requirements  defined  in  terL  of  statistical  availability  it  is  necessary 
that  design  concepts  which  yield  significant  improvements  in  maintainability, 
to  e^S,  ^ considered  tor  their  interactions  on  reliability  and  cost. 

A  primary  goal  of  a  designer  is  to  obtain  an  output  (pel 
will  satisfy  a  series  of  specific  performance  requirements .  Additional 
requirements,  such  as  reliability  and  maintainability  usually  appew'  as 
secondary  to  this  goal.  Moreover,  because  a  designer  usually  has  not been 
sufficiently  alerted  to  the  need  or  logic  of  such  requirements,  he  may 
view^them  as  burdensome,  if  not  irksome.  Under  the  stress  of  timepre.sures 
and  the  complexity  of  the  design  process  itself,  vhere  no  one  man  can  digest 
the  amount  of  specialized  technical  knowledge  and  implications 
vith  the  same  degree  of  insight  and  understanding,  reliability  and  maintain 
ability  requirements  are  most  likely  to  be  compromised. 

Finally,  the  neglect  of  maintainability,  serviceability,  reliability, 
and  producibiUty  frequently  »*ans  that  ultimate  system/equipment  schedules 
cannot  be  met  because  of  the  confujion  and  waste  caused  by  the  multiplicity 
of  engineering  changes  which  result  from  hasty  design. 
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Chapter  2 

BASIC  DESIGN  REVIEW  PHILOSOPHY 


Design  reviews  begin  vith  the  conceptual  phase  that  considers  the  broad 
general  requirements  and  as  the  design  approaches  the  hardware  stage,  narrows 
down  to  detailed  meetings  of  reduced  scope  where  only  circuits,  equipments, 
or  portions  of  equipments  are  considered,  and  then  broadens  again  as  the 
various  equipments  are  integrated  into  a  system. 

Design  changes  during  the  early- design  review  phases  generally  require 
very  little  engineering  effort  since  it  usually  involves  only  paper  changes 
of  a  part,  dimension,  or  value,  although  redesign  of  conponents  might  at 
times  be  mandatory.  Design  changes  occurring  during  subsequent  design 
reviews  involving  changes  to  drawings,  modifications,  or  replacement  of 
existing  hardware,  replacement  of  field  supplies,  revision  of  field  manuals, 
or  retraining  of  factory  and  field  personnel  for  example,  will  be  consider¬ 
ably  more  costly  ( 100-1000  times)  although  the  probability  of  such  changes 
will  be  less  than  during  the  first  phase.  As  it  pertains  to  reliability, 
maintainability,  value  engineering,  human  engineering,  etc.,  the  periodic 
review  of  design  at  key  points  in  the  development  program  facilitates 
detection  and  correction  of  actual  or  potential  design  problems  prior  to 
finalization  of  the  design. 

The  prime  purpose  of  a  formal  design  review  meeting  must  be  to  insure 
that  adequate  effort  has  been  made  by  the  designer.  The  design  review 
process  assures: 

a.  A  means  of  solving  interface  problems; 

b.  Confidence  that  experienced  personnel  are  involved  in  the 
design  detail; 

c.  A  record  of  why  decisions  were  made; 

d.  A  knowledge  that  systems  will  tie  together  and  be  compatible; 

e.  A  total  picture  for  the  benefit  and  use  of  the  final  decision¬ 
maker  In  making  trade-off  decisions;  and 

f .  A  greater  probability  of  a  fully  mature  design. 

A  design  review  plan  would  include  the  time-phased  events  representing 
the  appropriate  milestones  at  which  formal  system/ equipment  reviews  are 
made  at  major  decisions  points.  The  number  of  critical  decision  points 
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LEVEL  OF  REVIEW 


will  vary  according  to  the  type  of  development  program  underway.  The  broad 
categories  are  sometimes  listed  ss: 

s.  Conceptual  Design  Review. 

b.  Preliminary  Design  Review. 

c.  Preproduction  Design  Review. 

d.  Production  Design  Review. 


These  review  points  ore  keyed  to  major  events  and  consequently  reflect  the 
name  of  that  event.  It  is  well  to  bear  in  mind  the  objectives  of  these 
reviews  and  schedule  the  event  accordingly.  The  main  requirements  that 
con  fce  applied  to  any  program  will  be  covered  by  three  or  four  major  review 
pc  •  ry  s  ,  namely: 


a.  Conceptual  Design  Review. 

b.  Preliminary  Design  Review. 

c.  Detailed  Design  Review. 

d.  System  Design  Review. 


D#sigt  review  actions  could  be  likened  to  ar.  hour  glass  f  I  g“UT«  t he' 
first  considers  the  overall  concepts  that  narrows  dovrs  *s  msjer  decisions 
sre  reached  to  consider  the  e _ec trie* 1 /functional  desip  reviews ,  detailed 
hardware,  or  component  reviews;  then  'Dag: ns  to  broader,  to  combine  at  black 
ber  and  subeyrtem  level,  and  finally,  an  integrated  system  review.  (Gee 
Figure  below.]  Regardless  of  the  names  assigned  the  design  reviews, 
specific  mileaior.es  ar  decision  points  must  be  icaniif: ed  where  formal 
reviews  will  be  conducted. 


FIGURE  1 .  DESIGN  REVIEW  LEVELS 
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MXL-R-275^2,  paragraph  3 *5*10 >  defines  the  requirement  for  contractor 
engineering  design  reviews  for  reliability.  A  similar  requirement  for 
maintainability  is  found  in  MTL-M-26512,  paragraph  3.5.1.b.  Both  paragraphs 
require  the  submission  of  a  schedule  of  planned  reviews  to  a  procuring 
activity  and  permit  the  attendance  of  procuring  activity  personnel  at 
these  reviews . 

Since  ESD  policy  requires  the  submission  of  a  combined  plan  for  reli¬ 
ability  and  maintainability  and  since  system/equipment  requirements  are 
usually  defined  in  terms  of  statistical  availability,  it  is  expected  that 
separate  scheduling  of  reliability  and  maintainability  reviews  v Hi  not 
be  required.  Further,  for  good,  systems  engineering  practices,  system/ 
equipment  r  Ha'S  .  I :  iy /main  tainaM  lit y  engineering  reviews  cai  be  integrated 
within  the  genera]  framework  of  the  requirements  of  ATP  80-28,  Engineering 
Inspections.  However,  there  still  remains  the  need  for  the  depth  of 
inquiry  into  reHabtH ty  and  maintainability  engineering  design  si 
as  illustrated  by  the  Design  Review  Check  List  (see  attachment  l). 


5 


Chapter  3 

CONTRACTOR  DESIGN  REVIEW  BOARDS  IN  GENERAL 


where  required. 

The  organization  to  vhich  the  responsibility  for  design  review,  is 
assigned  will  vary  according  to  the  company  involved.  However,  the 

In  any  case,  the  ultlmte  responsibility  for  .design 
_«gi.  vlth  a  design  engineer,  hut  top  management  support  is  essential  rar 
S  t^a^ceed.  If  sound  recommendations  for  design  improvement 

£  £Z“  the  pr^am  Is  ZLd.  Although  it  nay  seldom  be  necessary, 
engineers* assigned  tHtsign  reviews  should  Know  that  they  can  appeal  to 
higher  management. 

A  Design  Review  Board  performing  design  reviews  may  be  coavoeei of 
nerraanent  members  augmented  by  experienced  talent  In  the  various  technical 
~  A  Design  Review  Board  should  have  one  or  more  senior  design 

effective  Board  should  be  limited  to  ten  members.  This 
will^vary  depending  upon  the  type  of  review  being  performedand  the^equ^p- 

to  the  complexity  of  the  system. 

bcperformed^by^nglneers^more  ^r^Zl  palpating 

?TfZ2  dasi^review^  ^^^Z%^eCd!^Z:u^cfaL.t^ 

tbey  sbould  be  scheduled 

“  .t7£cific  time  and  then  be  dismissed  „  soon  as  possible. 
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ESD  must  be  alert  to  contractor  methods  of  budgeting  for  scheduled 
design  reviews  to  assure  that  coats  are  not  compounded  by  each  department 
participating  in  reviews.  Design  review  costs  will  normally  be  pro¬ 
portional  to  the  complexity  of  the  equipment  which  dictates  the  number  of 
reviews  required  as  well  as  the  number  of  personnel  attending.  It  is 
important  to  stress  that  ESD  considers  design  reviews,  although  they  may 
appear  costly,  as  the  most  effective  means  of  assuring  that  the  Air  Force 
gets  a  full  measure  of  maturity  in  all  aspects  of  design.  A  rule-of-thumb 
figure  sometimes  applied  is  that  design  reviews  require  5^  of  the  overall 
design-manhours . 

Design  review  milestones  should  be  identified  early  in  the  program 
and  will  normally  be  coincident  with  the  main  development  phases  such  as 
conceptual,  breadboard  stage,  pre -prototype,  prototype,  preproduction, 
etc.  (see  Chapter  2).  The  review  points  identified  should  be  firmed  up 
approximately  30  days  in  advance  of  a  formal  design  review  and  data 
packages  should  be  distributed  to  all  attendees  along  with  lormal  notifi¬ 
cation  ten  days  prior  to  actual  date  a  review  is  to  be  held.  The  MIL-R- 
275^2  requires  the  contractor  to  notify  the  AF  procuring  agency  ten  days 
in  advance  of  a  meeting  so  they  may  participate  if  they  so  desire.  In 
any  case,  the  minutes,  agenda,  actions,  and  documentation  should  be 
available  for  review  when  requested  by  the  procuring  agency. 

Specific  information  that  must  be  reviewed  and  monitored  by  ESD  daring 
a  contractor  design  review  program  includes: 

a.  Personnel  (their  experience  levels)  assigned  to  the  program. 

b.  Organizational  assignments,  modus  operand!,  authority  delegated 
to  the  Design  Review  Board. 

c .  Design  handbooks  and  check  lists  prepared  for  -design  engineer¬ 
ing  use. 

d.  Design  review  plan- -mile 8 tone  identification,  etc. 

e.  Data  packages  developed  for  design  review  use.  These  packages 
should  include,  as  required,  worst  case  studies,  circuit  analysis,  parts 
application  data,  drawings,  etc.  The  completeness  of  packages  is  very 
important  for  individual  use  in  preparation  for  design  reviews. 

f .  Recorded  actions  by  a  Board  including  rejected  recommendations 
with  reasons  for  rejection. 

g.  Approved  design  changes  and  their  documentation. 

h.  Records  indicating  problem  areas  not  resolved  at  the  meeting 
with  action  assignments  for  resolution,  specific  problems  to  be  studied, 


target  dates  for  completion,  and  method*  of  follow-up  to  aaeure  completed 
actions . 

i.  Final  approval  of  design  by  respective  specialists  by  affixing 
signature  on  Board  minutes. 


Chapter  U 

CONCEPTUAL  DESIGN  REVIEW 


The  conceptual  design  review  Is  the  most  important  design  review  to  be 
accomplished.  Important  decisions  are  made  at  this  time  that  preclude  or 
freeze  subsequent  designs.  It  Is  therefore  logical  that  this  review  should 
be  attended  by  the  largest  group  of  knowledgeable  engineers.  This  Design 
Review  Board  must  consider  the  feasibility  of  design;  the  techniques  to 
be  employed  In  achieving  performance  requirements ;  the  interface  problems 
which  involve  system,  maintenance,  and  design  concepts;  and  specific  design 
requirements  that  might  conceivably  push  the  state-of-the-art.  Major 
design  characteristics  such  as  performance,  reliability,  maintainability, 
and  value  must  be  carefully  considered.  A  proposed  configuration  should 
be  reviewed  for  such  considerations  as  the  use  of  standard  circuits  of 
proven  reliability,  comparison  of  one  computer  manufacturer  with  another, 
evaluation  of  be It- drive  versus  direct-drive,  the  need  for  redundant 
replacements ,  the  hardware  approach  to  be  followed  in  the  Identification 
and  localization  of  system  failures,  methods  to  minimize  the  influence  of 
limited  or  critical  life  items  on  the  operational  capability  of  the  system/ 
equipment,  etc.  With  such  considerations  a  paper  study'  may  be  accomplished 
to  obtain  an  estimate  of  the  system’s  reliability,  maintainability,  or 
other  figures  of  merit.  This  study  is  then  available  as  a  valuable  tool 
to  assist  the  Board  In  selecting  the  ultimate  system  configuration. 

Although  early  reviews  cannot  be  rigorous  in  design  detail,  the  early  design 
decisions  are  extremely  important  for  these  decisions  commit  the  program 
to  a  sp*  rifle  design  approach  or  strategy.  Improper  logic  or  design 
approaches  shoiULd  be  ferreted  out  at  this  point  while  changes  involve  only 
paper  changes  and  before  actual  equipments  begin  to  take  shape.  As  the 
design  progresses,  subsequent  changes  become  much  more  expensive  and  tedious 
to  accomplish  (see  Chapter  2) . 

The  material  or  data  that  should  be  available  for  use  by  the  Board  In 
Its  preparation  and  deliberation  includes: 

a.  The  proposal; 

b.  The  Specific  Operational  Requirement  (SOR); 

c.  The  Statement  of  Work  and  associated  specifications; 

d.  The  analysis  of  system  requirements; 

e.  Baaic  design  criteria  (block  or  logic  diagrams  and  flow 

charts ) ; 

f.  Reliability  and  maintainability  requirements; 
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g.  Possible  trade-off  documentation;  and 
b.  System/equlpment  schedules  vith  milestones. 

An  important  outcome  of  conceptual  or  system  design  reviews  is  the 
ability  of  the  systems  contractor  to  quantify  reliability  and  maintainability 
requirements  at  the  subsystem  level  for  the  guidance  of  design  engineering 
personnel  and  for  insertion  into  subcontracted  equipment  specifications. 

Subsystems  and  component  detail  design  reviews  are  concerned  vith 
determining  the  maintainability  and  reliability  characteristics  of  subsystems 
during  the  detail  design  phase  of  program  development.  The  purposes  of 
this  effort,  described  in  Chapter  5,  are  to  determine  the  extent  to  which 
the  various  designs  in  process  will  achieve  the  requirements  set  forth  as 
the  result  of  the  conceptual  or  system  design  review  and  to  indicate  the 
need  for  redistribution  of  system  requirements. 


Chapter  5 

DETAILED  SUBSYSTEM  DESIGN  REVIEWS 


The  number  of  formal,  detailed  subsystem  design  reviews  scheduled  and 
the  optimum  time  for  reviews  will  vary  as  a  function  of  the  system  complex¬ 
ity,  the  type  of  equipment  being  utilized,  the  caliber  of  cognizant  design 
engineers,  etc.  However,  formal  design  reviews  should  be  conducted  prior 
to  release  of  any  design  to  production.  At  major  review  points  every 
facet  of  the  design  considerations  should  be  cdref ally  gone  over.  A 
design  review  check  list  should  be  utilized  to  assure  consideration  of  all 
important  criteria  (see  attachment  l) .  A  check  list  may  by  necessity  be 
tailored  to  fit  the  specific  requirements  of  a  system,  but  in  any  case  it 
should  not  be  a  different  set  of  criteria  from  the  ones  used  by  designers. 
It  would  be  unreasonable  to  confront  the  designer  with  a  new  set  of  rules 
at  the  time  of  review. 

A  design  handbook  should  be  prepared  to  reflect  the  specific  require¬ 
ments  of  the  project  and  mads  available  to  design  engineers.  These  design 
handbooks  should  be  reviewed  for  adequacy  of  content  and  acceptability  to 
ESD  programs  prior  to  coosnencing  the  design  effort.  ESD  does  not  presume 
to  dictate  the  manner  in  which  a  design  review  will  be  conducted,  but 
aims  to  evaluate  that  effort  and  to  take  appropriate  action  when  review 
actions  fail  to  satisfy  the  design  review  purpose  as  related  to  scope, 
depth  of  analysis,  corrective  actions  taken,  or  experience  of  participating 
members . 

Development  Engineering  personnel  of  the  Contract  Management  Regions 
should  be  fully  utilized  to  provide  continuous  surveillance  of  .the  design 
review  effort.  This  source  of  engineering  talent  is  iinportaat  to  the  SPO 
effort  and  should  not  be  overlooked — their  contribution  will  be  of  great 
value  to  the  overall  effort. 

Design  reviews  should  not  be  staged  affairs  that  reflect  the  results 
of  previous  meetings,  but  should  indicate  a  thorou^i  preparation  and 
attention  to  detail  by  all  participants .  The  design  engineer  should  be 
prepared  to  defend  all  decisions  reached  by  him  by  presenting  required 
studies  (including  breadboard  test  data,  if  available),  and  engineering 
calculations.  He  should  be  prepared  to  defend  the  selection  of  a  resistor, 
for  example,  not  by  merely  stating  that  it  is  reliable,  but  by  saying 
this  resistor  was  chosen  because  it  is  a  standard  item  with  the  lowest 
possible  cost  to  perform  the  required  function;  it  is  derated  to  25 of 
its  normal  rating  for  the  following  reasons  ...  ;  it  is  considered  as 
reliable  as  any  item  available  baaed  on  the  present  state-of-the-art  and 
is  expected  to  give  a  long  trouble-free  life  or  MTEF  of  X  hours . 
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Engineers  attending  reviews  should  be  thorough  in  their  pre- review 
analysis  of  what  they  consider  as  potential  problem  areas  and  should  be 
prepared  to  indicate  in  detail  what  the  effects  of  their  recommended 
changes  will  have  upon  the  major  characteristics  of  the  equipment.  It 
is  also  important  that  Board  members  notify  the  designer  of  areas  of 
disagreement  in  sufficient  time  before  the  formal  meeting  to  allow  him  to 
assemble  reference  material  to  support  his  decisions;  thus  allowing  the 
Bsview  Board  to  thoroughly  consider  both  sides  cf  the  question.  The 
complste  analysis  and  presentation  of  facts  rather  them  theories  enables 
sound  decisions  to  be  reached  in  the  shortest  period  of  tire.  Examples  of 
the  types  of  data  necessary  to  facilitate  detailed  reviews  include: 

a.  System  reliability  predictions. 

b.  Detailed  subsystem;  circuit  reliability  predictions. 

c.  Maintainability  predictions;  studies,  and  task  simulation 

results . 

d.  Component  parts  lists  with  appropriate  test  information. 

e.  Farts  derating  and  application  data. 

f.  Farts  failure  rate  data  (or  sources). 

g.  Stress  analysis  results. 

h.  Failure  effects  analysis. 

i.  Statistical  analysis  of  circuit  (or  assembly)  performance  as 
a  function  of  parts  variability .  Error  and  tolerance  studies . 

J.  Reliability  aspect  of  redundant  parte,  assemblies,  subsystems, 
modes  of  operation  with  attention  to  switching  problems. 

k.  Consideration  of  potential  reliability  growth. 

l.  Documented  reliability  growth  plans. 

m.  Analyses  of  known  trouble  areas,  with  plans  for  corrective 

action • 

n.  Technical  data,  including  equipment  physical  construction  and 
profiles,  block  diagrams,  schematics,  signal  flow  charts,  equipment 
operating  theory,  maintenance  philosophy,  operating  procedures  and  main¬ 
tenance  instructions . 
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The  maintainability  portion  of  design  reviews  are  concerned  with  such 
equipment  or  design  features  as: 

a.  Packaging  -  Is  equipment  of  modular  construction  for  easy  unit 
replacement? 

b.  Labeling  -  Are  parts  and  controls  clearly  and  accurately 

labeled? 

c.  Protective  devices  for  components  and  circuits. 

d.  Quick-release  features  of  connectors,  latches,  and  fasteners. 

e.  Availability  and  access  of  test  points. 

^  f  •  Self- test  features  or  built-in  test  equipment .  Operation  and 

fault  detection  features. 

g.  Adjustments  -  Arc  required  adjustments  kept  to  a  minimum? 

While  the  above  design  features  are  important  for  the  achievement  of 
quantitative  parameters,  it  is  recognized  that  maintainability  improvement 
;s  not  United  equipment  changes,  but  can  be  produced  by  providing  proper 
diagnostic  routines  and  maintenance  aids  which  increase  relative  ease  and 
simplicity  cf  performing  maintenance  tasks,  changes  in  operating,  test  o~ 
maintenance  procedures,  and  insuring  the  efficient  selection  and  utilization 
of  tooxs,  test  equipment  and  maintenance  personnel.  Inherent  maintainability 
or  re pa  -ability  is  a  design  feature  which  can  be  controlled  during  the 
desip-,  process,  but  the  achievement  cf  satisfactory  operational  maintainability 
requires  a  consideration  of  a  wider  spectrum  of  activities  than  the  desicr 
process  itself.  & 

As  the  design  process  progresses  to  the  stage  where  mock-ups  are  available 
an  efficient  and  excellent  design  review  tool  is  available  to  a  Design  Review 
Board,  maintainability  engineer  can  make  maximum  use  of  mock-ups  in 

visualizing  access  requirements,  resolving  space  conflicts,  etc.  The  ability 
to  visualize  a  problem  in  a  realistic  three  dimensional  environment  can  serve 
to  expedite  the  arrival  at  satisfactory  solutions  to  problems.  'Where  problems 
arise  that  involve  making  trade-offs,  the  mock-up  should  enable  personnel 
working  on  problems  to  visualize  various  alternative  solutions  before  drawing 
board  time  is  expanded.  Unsatisfactory  trade-offs  can  be  discarded  before 
any  appreciable  expense  has  been  incurred  in  exploring  them. 

An  important  inherent  consideration  of  the  engineering  design  review 
process  is  the  identification  of  preventive  and/or  corrective  maintenance 
tasks  which  require  actual  demonstration.  Task  demonstration  requirements 
Wiil  most  frequently  be  generated  as  a  result  of  one  or  more  of  the  following 
considerations: 

a.  Tasks  are  highly  critical  in  terms  of  system/equipment  operability 
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b .  Tasks  are  unique,  exceedingly  exacting  and/or  new  or  complex. 

c.  Tasks  are  associated  with  potentially  hazardous  environments 
and/or  equipments  and  adequacy  of  safety  procedures  and  devices  needs  to 
be  verified  by  simulation  before  actual  task  performance  can  safely  be 
permitted. 

The  selection  of  taste  in  turn  requires  that  a  demonstration  procedure 
be  established  for  each  task.  The  procedure  must  specify  precisely  what 
is  to  be  done,  the  tools,  equipment  and  personnel  to  be  employed  in  per¬ 
forming  the  task,  the  environment  and  equipment  configuration  to  exist 
during  the  demonstration  and  an  explanation  of  exactly  what  specific 
information  is  expected  to  be  obtained.  The  results  of  task  demonstrations 
should  become  a  part  of  the  technical  data  available  to  a  Design  Review 
Board.  Simulation  results  may  uncover  features  of  equipment  design 
requiring  corrective  action. 

Finally,  while  the  discussion  has  been  on  formal  Design  Review  Board 
actions,  ESD  should  take  a  more  general  look  at  the  interaction  of  a 
contractor's  reliability/maintainability  and  design  organizations  during 
program  monitoring.  The  influence  of  the  reliability/maintainability 
organization  on  the  design  process  must  not  be  felt  only  at  formal  Board 
meetings.  A  continuous  interaction  on  questions  of  design  strategy  should 
take  place  between  these  organizations  throughout  the  design  process . 


Chapter  6 

DESIGN  REVIEW  CHECK  LIST 


The  design  process  itself  and  subsequent  reviews  should  never  be  left 
to  chance,  but  should  always  be  conducted  according  to  a  systematic  plan. 

To  assure  that  important  design  considerations  have  been  considered  by 
designer  and  reviewer,  a  ccnprehensive  check  list  should  be  employed. 

An  example  of  a  list  developed  by  Radio  Corporation  of  America  (RCA) 
is  presented  for  general  guidance  in  Attachment  1.  The  material  in  the 
check  list  presents  a  formidable  array  of  questions  which  have  been  found 
necessary  for  an  adequate  evaluation  of  system/equipment  characteristics. 

While  the  check  list  serves  as  a  design,  it  is  important  to  recall 
that  design  guidelines  are  available  in  numerous  publications  by  Government 
agencies  ;  for  example,  the  RADC  Reliability  Notebook  contains  a  family  of 
interaction  models  which  relate,  for  various  part  classes,  part  operating 
stresses  and  failure  rates  and  can  be  applied  by  a  designer  in  arriving  at 
satisfactory  component  port  derating  procedures  (subject  to  the  constraints 
of  size,  weight,  and  cost). 

ESD  personnel  attending  formal  design  review  meetings  are  also  urged 
to  become  familiar  with  the  design  techniques  described  in  various  AFSC 
manuals  to  assist  in  their  evaluation  of  proposed  designs.  Several  methods 
of  circuit  analysis  for  reliability  purposes  are  described  in  Attachment  2. 


DESIGN  REVIEW  CHECK  LIST 


PART  X  7  EMZPftAL 

1 .  Parts  Selection  and  Evaluation 

a.  Have  the  appropriate  standards  been  consulted  for  selection 
of  standard  electrical  components? 

b.  Can  a  redesign  omit  a  nonstandard  part  or  replace  it  with  a 
standard  part? 

c.  What  parts  are  nonstandard? 

d.  Hava  raquests  been  initiated  for  approval  of  nonstandard 
par  t  s  ? 

e.  Have  environmental  tests  been  started  on  nonstandard  parts? 

f.  Have  potted  circuits  been  subjected  to  environmental  testing? 

g.  What  are  the  parts  having  the  highest  failure  rates? 

2.  Parts  Application 

a .  Resistors 

(1)  What  is  the  operating  ambient  temperature? 

(2)  What  power  dissipation  Is  estimated  in  this  application? 

(3)  Is  the  resistor  properly  derated? 

(4)  What  tolerance  limit  is  required  for  satisfactory  circuit 
operat  ion? 

(5)  What  tolerance  buildup  (due  to  temperature,  aging, 
electrical  stress,  etc.)  can  be  allowed? 

(6)  Has  the  rated  wattage  been  adjusted  in  cases  where  short 
mounting  leads  are  used? 

(7)  Can  any  potentiometers  be  replaced  by  resistors? 

(8)  Has  the  voltage  limit  been  excaedad  on  any  fixed  composi¬ 
tion  resistors? 

b.  Capacitors 

(1)  What  is  the  operating  ambient  temperature? 
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(2)  What  is  the  working  voltage  expected  in  this  application? 

(3)  Is  the  capacitor  properly  derated? 

(4)  Is  tha  capacitor  sub j act  to  surge  voltages  which  excead 
tha  rated  operating  voltage? 

(5)  Vhat  tolaranca  limit  is  required  for  satisfactory  circuit 
operation? 

(6)  Vhat  tolar ance  buildup  can  be  allowed? 

(7)  Vhat  derating  factor  was  used  for  a-c  ripple  or  pulse 
voltages  on  MIL-E-25A  paper  capacitors? 

(8)  Hava  capacitors  with  adaquata  temperature  ratings  bean 
used  wharever  possible? 

(9)  Hava  tamper atur© -compensating  or  low  temperature  coefficient 
capacitors  (mica  or  ceramic)  been  usad  wharavar  high 
stability  is  required? 

(10)  Hava  high  dialactric  caramic  cepacitors  been  restricted 
to  bypass  usage? 

(11)  Are  tantalum  capacitors  bypassed  for  high  frequencies 
(above  100  kc)  7 

(12)  Are  all  capacitors  heavier  than  0.5  oz.  securely  mounted 
in  accordance  with  specification  MIL-E-5400,  para. 

3. 1.3. 5? 

c.  Tubes 

(1)  Does  tha  specification  of  tha  tube  type  selected  define 
tha  required  characteristics? 

(2)  Does  tha  operation  of  the  tube  approach  any  absolute 
rating  under  any  usual  variation  of  supply  voltage  or 
load? 

(3)  Vhat  is  the  operating  ambient  tempera tura? 

(4)  Vhat  electrode  ratings  era  of  critical  consideration  in 
this  circuit  application? 

(5)  Is  the  heater  voltage  within  rating?  Vhat  variations 
era  expected? 
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Is  the  haater-to-cathoda  voltage  within  rating  in  this 
circuit  application? 

(7)  Are  the  plate  and  screen  grids  properly  derated? 

(8)  Vhat  tolaranca  buildup  can  be  allowed? 

(9)  Has  Gm  variation  bean  considered? 

(10)  Were  maximum  grid  resistance  ratings  observed? 

(11)  Is  input  and/or  output  capacity  a  critical  considaration 
in  this  circuit  application?  Vhet  variation  in  input 
and/or  output  capacity  can  be  toleretad? 

(12)  Does  circuit  operation  dapend  upon  e  tube  parameter  not 
controllable  by  the  designer? 

(13)  Vhat  is  the  maximum  rated  vs.  maximum  expected  bulb 
temperature? 

(14)  Will  the  circuit  perform  satisfactorily  with  randomly 
selected  tubes?  -  with  tubes  operating  et  thair  upper  or 
lower  MIL  limits? 

(15)  Has  tube  approval  dete  been  taken? 

(16)  If  e.  printed-circuit  board  is  being  used,  have  adequate 
cooling  measures  (convection  to  cooling  air  or  conduction 
to  e  haet  aink)  bean  taken  to  prevent  damage  to  the  board 
or  components  mounted  on  it? 

(17)  Have  standard  tube  shialds  bean  used? 

Transistors 

(1)  Does  the  specification  of  tha  type  of  transistor  selected 
defina  the  required  characteristics? 

(2)  Doaa  the  operation  of  tha  transistor  approach  any  abso¬ 
lute  rating  under  any  usual  variation  of  supply  voltage 
or  load  ? 

(3)  Vhat  is  tha  operating  ambient  temperature? 

(4)  Vhat  ia  tha  maximum  rated  power  dissipation?  Vhat  is 
the  maximum  power  dissipation  expected  in  this  circuit 
application? 
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(5)  Vhat  is  tha  maximum  rated  collector  voltage?  Vhat  ia 
the  maximum  collector  voltage  in  tha  preeent  application? 

(6)  Vhat  ia  the  maximum  rated  collector  current? 

(7)  Vbet  deviation  in  Beta  ie  tolereble? 

(8)  Hov  much  deviation  in  Beta  is  expected  due  to  tolerance 
buildups? 

(9)  Will  the  circuit  perform  setisfectorily  with  rendomly 
selected  treneistors?  -  vith  trensistora  operating  et 
their  upper  or  lover  MIL  limits? 

(10)  Is  power  gain  a  critical  consideration  in  this  applica¬ 
tion? 

(11)  Vhat  deviation  in  power  gain  is  tolerable? 

(12)  Vhat  deviation  in  power  gain  is  expected  due  to  tolerance 
buildup? 

(13)  Is  noise  figure  e  critical  consideration  in  this  applica¬ 
tion? 

(14)  Is  the  noise  figure  tolereble  at  the  operating  ambient 
temperature  ? 

(15)  Hov  much  leakage  current  is  expected  et  the  operating 
ambient  temperature? 

e .  Semi conduc tor  Diodes 

(1)  Does  the  specificetion  for  the  type  of  diode  selected 
define  the  required  characteristics? 

(2)  Vhet  ie  the  operating  ambient  temperature  for  each 
dioda? 

(3)  Vhet  ie  the  power  dissipation  within  the  diode?  Vhat 
ie  the  maximum  rated  power  diaaipation? 

(4)  Hov  much  reverse  recovery  time  doea  the  diode  require? 

(5)  Vhat  ia  the  reted  peak  inveraa  voltage? 

(6)  Hov  much  ravaraa  current  can  be  tolerated? 

(7)  Hov  Batch  reverse  current  will  flow  at  the  operating 
ambiant  tamperature? 
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(8)  Does  tha  circuit  perform  satisfactorily  vith  randomly 
selected  diodee?  -  vith  diodes  operating  et  their  upper 
or  lover  MIL  limits? 

(9)  Vhat  Zenar  voltage  reference  is  required?  Vhet  Zener 
raference  voltage  i3  expected? 

f.  Transformers.  Chokes  end  Coils 

(1)  Whet  ie  the  operating  ambient  tempereture? 

(2)  Is  Q  e  critical  consideration  in  this  circuit  applice- 
tion?  Vhat  deviation  in  Q  can  be  tolereted? 

(3)  Vhet  deviation  in  Q  is  expected  due  to  tolerence  buildup 
end  to  temperature  changes? 

(4)  Vhet  is  the  maximum  current  carrying  cepe bill ty  of  the 
choke  or  coil?  Vhet  is  the  maximum  current  expected 
in  this  application? 

(5)  Hov  close  is  the  highest  operating  frequency  to  the 
reeonant  frequency  of  the  choke  or  coil? 

(6)  Has  a  requirement  for  ehielding  been  established? 

(7)  When  a  hum.  problem  exists,  has  special  consideration 
been  giver  to  core  construction? 

(8)  Do  transformer  specificetions  conform  to  MIL  s+anderds? 

g,  Pelave  and  Svltchee 

(1)  Vhet  "quality  level”  does  eech  relay  or  switch  represent? 

(2)  Hov  many  actuations  per  hour  are  expected? 

(3)  Hov  meny  actuations  per  mission  are  expected? 

(4)  Vhat  percent  of  reted  current  does  eech  contact  cerry? 

(5)  Is  reley  closing  time  or  opening  time  ■  crj'icrl  considere- 
tion?  If  ao,  hov  much  increaee  is  tolerable? 

(6)  Vhat  are  the  pull-in  end  dropout  voltages  or  currents? 

(7)  Vhet  is  the  manufacturer 'e  tolerence  for  initial  coil 
rasietanca. 

(8)  Hov  much  will  the  coil  resistance  vary  vith  tempereture? 
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(9)  How  much  chenge  in  coil  resistance  is  tolerable  • 

(10)  Has  arc  auppresaion  been  used? 

(11)  Has  tba  possibility  of  dry  circuit  operation  bean 
considered? 

b.  Electromechanical  Devices 

(1)  Heve  the  adverse  affects  on  bruahaa  at  high  altitudes 
been  considered? 

(2)  What  consideration  has  bean  given  to  variations  of  d-c 
motor  apeed-torque  characteristics  due  to  temperature 
and  eltitude? 

(3)  How  critical  to  proper  operation  is  tha  speed-torque 
characteristic? 

(4)  Can  tha  associated  circuitry  tolerate  increased  loads 
caused  by  variation  in  motor  characteristic? 

(5)  Heve  tha  eppropriate  specialists  bean  consulted  on  the 
use  of  rotary  solenoids  and  timing  motors? 

(6)  Have  you  depended  solely  on  manufacturer 's  data  for  force 
movement  characteristics  of  solanoida? 

(7)  Ara  metar  windows  sealed  to  prevent  moiature  formation? 

(8)  Haa  the  possibility  of  charge  formation  on  meter  windows 
bean  inveatigeted? 

(9)  Are  rasolvars  checked  for  accuracy  and  phese  shift  at 
elevated  temperatures? 

i.  Connector a  and  Plugs 

(1)  Does  the  number  of  active  pins  per  connector  conform  to 
tha  recommanded  limit? 

(2)  Is  a  sufficient  number  of  spare  pins  available  on  each 
connector?  (At  lasst  four  spares  for  connectors  over 
26  pins  per  MIL-E-5400C,  paragraph  3. 1.5. 3.) 

j.  Miscellaneous  Partj  (Printed  Circuits,  Vira,  etc J 

(l)  Has  consideration  been  given  to  tha  currant  rating  of 
wire? 
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(2)  Has  the  current  rating  of  wire  bean  reduced  in  cases 
where  voltage  drop  is  important? 

(3)  Is  vira  color  coding  required,  and,  if  ao,  is  it  in 
accordance  with  the  proper  atanderds  or  specifications? 

(4)  Haa  the  placement  of  components  on  print ad -circuit 
boards  been  considered  from  the  cross-talk  point  of 
view? 

(5)  Does  e  beat  dissipation  problem  axist  on  printed -circuit 
boards? 

(6)  Is  e  keying  achema  employed  to  prevent  interchenging 
printed -circuit  boerds? 

(7)  Are  transistor,  dioda,  and  tentalytic  capacitors  properly 
polarized  on  print  ad -circuit  boerds? 

(8)  Are  large  potential  gradienta  posaible  between  adjacent 
pins  or  connectors  on  printed-circuit  boards? 

(9)  Do  circuit  breakers  conform  to  MIL-C-5989B? 

3.  System  end  Circuit  Considerations 

(1)  What  variations  in  input  signel  can  be  tolerated?  Whet 
variations  ara  expected? 

(2)  Whet  variations  in  the  impedance  presented  to  the  input 
terminals  can  be  tolerated?  What  is  expected? 

(3)  How  does  the  input  circuitry  contribute  to  input  tolerances? 

(4)  la  a-c  power  supply  distortion  a  critical  consideration? 

(5)  What  percentage  of  distortion  can  be  tolereted?  Whet  is 
expected? 

(6)  Whet  tests  have  been  performed  to  confirm  the  ensvers  to 
questions  in  Para.  (5)  above? 

(7)  What  variation  in  B+  voltage(a)  can  be  tolerated?- 

(8)  What  variatior  in  bias  voltage  can  be  tolerated? 

(9)  What  design  features  protact  the  circuit  ageinst  excessive 
variations  in  line  voltage?. 
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(10)  Whet  design  features  protect  the  circuit  against  loss  of  ri* 
or  bias  voltage  aupplies? 


(11)  What  cable  length  was  assumed  on  inputs  and  outputs 

(12)  How  much  change  in  the  assumed  cable  length  can  be  tolerated. 

(13)  Is  over-sll  protection  provided  against  overload,  excessive 
heating,  pressure  changes,  etc; 


(U) 

(15) 

(16) 


Do  the  self-test  features  of  the  unit  meet  the  requirements? 


What  problems  were  observed  when  the  circuit  was  tested  in 
conjunction  with  other  units? 


Has  the  unit  been  subjected  to  er.vironx 
problems  were  observed  with  respect  to 
vibration,  shock,  altitude? 


nrdrl  icrlinp?  What 
temperature,  moisture, 


(17)  Have  ell  problems  highlighted  in  the  preliminary  design 
review  been  resolved? 


(18)  Has  a  separate  list  of  recommendations  for  product  improve¬ 
ment  or  redesign  beer  compiled? 

(19)  What  alternate  circuits  or  systems  were  considered; 

(20)  Have  "preferred  circuits"  been  used  wherever  possible? 

(21)  What  factors  influenced  the  choice  of  tbis  particular 
circuit  or  system? 

(22)  Are  there  firm  specifications  for  this  circuit,  including 
test  specifications? 

(23)  Have  all  specifications  been  met  unconditionally ? 

(24)  Does  any  specification  require  modification. 

(25)  Can  any  unreasonable  or  unusually  difficult  requirement 
be  relaxed? 


(26)  Can  a  simulation  study  be  of  assistance? 

(27)  What  marginal  testing  has  been  performed?  Wes  marginal 
operation  indicated  in  any  case?  What  are  tbe  critical 
parameters  affecting  marginal  operetion? 
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(23)  Have  heat  runs  been  made  on  electrical  components  which  are 
either  thermal  emitters  or  otherviee  heat  sensitive? 

(29)  Have  phase  margin  checks  been  performed  on  all  feedback 
loops? 

(30)  What  decoupling  or  neutralization  echemes  have  been  imple¬ 
mented  to  avoid  regenerative  feedback  loops? 

(31)  What  analyses  have  been  performed  to  determine  the  existence 
of  feedback  loops  and  their  effects  on  other  circuits? 

(32)  Is  circuit  operation  contingent  upon  the  proper  positioning 

of  more  than  one  switch  or  control;  i.e.,  are  several  adjustable 
components  necessary  in  the  circuit? 

(33)  Can  any  circuits  be  simplified  and  still  operate  within  require¬ 
ments?  (On  a  value  improvement  basis) 

(34)  Is  the  unit  capable  of  satisfactory  operation  after  tbe  mini¬ 
mum  required  warm-up  time? 

(35)  What  system  adjustments  are  required  when  a  unit  is  replaced? 

(36)  What  means  are  employed  to  decouple  the  power  supply? 

(37)  Do  parasitic  oscillations  exist? 

(38)  What  design  features  have  been  incorporated  to  suppresa 
parasitic  oscillations? 

(39)  What  are  the  required  tolerances  on  output  signals?  What 
ere  the  expected  variations? 

(40)  How  does  the  circuitry  contribute  to  output  tolerances? 

(41)  Do  weight  reduction  considerations  affect  reliability? 

(42)  Have  static  and  dynamic  power  drains  been  determined? 

4.  Reliability  Analysis 

(1)  What  is  the  estimated  required  mean  life  of  this  circuit? 

(2)  What  is  the  calculated  mean  life? 

(3)  Wbat  is  the  mean  life,  based  on  bench  or  other  teete? 

(4)  Is  there  a  history  or  record  of  bench  failuree? 
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(5)  Hava  random  failure  ratea  and  vearout  rates  been  eatebliahed 
for  all  parta? 

(6)  What  parte  hava  an  excessive  failure  rate? 

(7)  What  assumptions  were  made  in  calculations  with  raapect  to 
dareting  end  temperetura? 

(8)  Are  any  parta  opereting  near  or  above  their  recommended 
ratings? 

(9)  Has  a  statiaticel  analysis  been  conducted  to  determine  effects 
of  drift  in  component  parameters  end  of  component  tolerance 
buildups? 

(10)  Hnc  :«  fail-safe  design  philosophy  bean  utilized? 

(11)  Is  protection  ageinat  secondary  failures  (resulting  from 
primary  failure  a)  incorporated  where  possible? 

5.  Safety  Factors 

(1)  Is  there  adequate  protection  against  dangerous  voltages? 

(2)  Are  high-voltaga  warning  plataa  necessary? 

(3)  Hava  interlocks,  safety  switches  and  grounding  bars  been 
considered? 

(4)  Ara  all  external  metal  parta  at  ground  potentiel? 

(5)  Ara  discharging  rods  necessary  for  large  capacitors?  (at 
least  10,000  ohms) 

(t)  Are  bleeder  and  current  limiting  reaialors  uaed  in  power 

supplies? 

(7)  Are  there  burning  hazards? 

(8)  Are  "hot"  terminals  exposed  when  plugs  or  connectors  are 
not  connected? 

(9)  Ara  adjacent  plugs  or  connectors  keyed  to  pravant  inter¬ 
changing  connections? 

(10)  Can  maintenance  or  adjustment  be  performed  safely? 
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(1)  Are  the  maintenance  and  test  equipment  requirements  compatible 
with  the  concept  established  for  the  system? 

(2)  Does  the  unit  require  tipeciel  handling? 

(3)  Can  the  unit  be  readily  installed  end  connected  to  the  system? 

(4)  Are  factory  edjustments  such  that  they  do  not  require  readjust¬ 
ment  whan  units  ara  replaced  in  a  system  or  when  parts  era  re¬ 
placed  in  the  unit  in  the  field? 

(5)  Whet  adjustments  ara  necessary  after  a  unit  has  been  installed 
in  the  system? 

(6)  Ara  adjustments  capable  of  compensating  for  all  possible 
tolaranca  bull dupe? 

(7)  Is  periodic  alignment  and/or  adjustment  recommended?  How 
often? 

(8)  Ara  all  requirements  for  maintenance  teats  such  that  the 
specified  time  limitations  can  be  met? 

(9)  Has  tha  number  of  factory  adjustments  been  minimized? 

(10)  Has  tha  number  of  field  adjustments  been  minimized? 

( 1 1 )  Are  interconnected  circuits  in  the  same  package,  thus  pro¬ 
viding  minimal  inputs  and  outputs  et  each  maintenance  level? 

(12)  Is  tha  interaction  between  nd iustmants  and  other  circuit 
parameters  minimized? 

(13)  Is  tha  design  such  that  damage  to  tha  circuit  cannot  result 
from  caralass  usa  of  en  edjustment  or  combination  of 

ed justments? 

(14)  Ara  all  adjustments  and  indicators  of  tha  "cantar  zero"  type 
where  possible? 

(15)  Is  periodic  testing  necessary?  How  often? 

(16)  Are  the  test  points  ed equate?  Ara  they  accessible  in  tha 
installed  condition? 

(17)  Whet  overhaul  tasting  is  required? 

(18)  Whet  specific  test  equipment  is  necessary? 
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(19)  Have  factory  and  maintenance  teet  equipment  requiremcn  : 
been  r.Jniri7ed  and  coordinated  with  the  requirements  for 
other  units? 

(20)  What  special  techniques  ere  required  in  the  repair,  replace¬ 
ment,  or  alignment  of  the  unit? 

(21)  Are  parts,  assemblies,  and  components  pieced  so  there  is 
sufficient  space  to  use  test  probes,  soldering  iron,  and  other 
tools  without  difficulty?  Are  they  placed  so  that  structural 
members  of  units  do  not  prevent  access  to  them? 

(22)  Are  testing,  alignment  and  repair  procedures  such  thet  a  min¬ 
imum  of  knowledge  is  required  on  the  pert  of  maintenance 
personnel?  Can  trouble  shooting  of  an  assembly  take  place 
without  removing  it  from  e  major  component? 

(23)  Vhat  special  tools  and/or  test  equipment  are  required? 

(24)  Can  every  fault  (degrading  or  catastrophic)  which  cen  possibly 
occur  in  the  unit  be  detected  by  the  use  of  the  proposed  test 
equipment  and  standard  teet  procedures? 

(25)  Keve  parts  subject  to  early  weerout  been  identified?  Have 
suitable  preventive  meir.tenence  schedules  been  esteblished 
to  control  these  parts? 

(26)  Are  the  components  having  the  highest  failure  rates  reedily 
accessible  for  replacement? 

(2'7)  Are  parts  mounted  directly  on  the  mounting  structure  rether 
then  being  stacked  one  on  another? 

(2?)  Are  units  and  assemblies  mounted  eo  thet  replacement  of  one 
does  not  require  removel  of  others? 

(29)  Are  limiting  resistors  used  in  teat  point  circuitry;  i.e., 
is  any  component  likely  to  fail  if  a  test  point  is  grounded? 

(30)  Can  panel  lights  be  easily  replaced?  (Panel  lights  should 
rot  be  wired  in  series) 

(31)  Have  voltage  dividers  been  provided  for  test  points  for 
circuits  carrying  more  than  300  volte? 

(32)  Will  the  circuit  tolerete  the  uae  of  a  Jumper  ceble  during 
maintenance? 


Attachment  1 
Page  12  of  2 6 


2b 


u  -  •  .M&fe 

-Vj 

(33)  Are  controls  loceted  where  they  cen  be  eeen  and  opereted 
without  disassembly  or  removal  of  any  pert  of  the 
installation? 

(34)  Are  releted  displays  and  controle  on  the  eame  fece  of  the 
equipment  ? 

(35)  Are  all  units  (and  parte,  if  possible)  labeled  with  full 
identifying  data?  Are  parts  stamped  with  relevant  electricel 
characteristics  information? 

(36)  Are  cables  long  enough  to  permit  each  functioning  unit  to  be 
checked  in  a  convenient  place? 

O'7)  Are  plugs  and  receptacles  used  for  connecting  cebles  to 
equipment  units,  rether  than  "pigtailing"  to  terminal 
blocks? 

(3«)  Are  field  -replaceable  modules,  parts  and  subessemhilies 
plug-in  rether  then  soldered? 

(39)  Are  cable  harnesses  designed  for  fabrication  es  a  unit  in 
a  shop? 

(40)  Are  cebles  routed  to  preclude  pinching  by  doors,  covers,  etc? 

(41)  Is  eech  pin  on  each  plug  identified? 

(42)  Are  plugs  designed  to  preclude  insertion  in  the  wrong 
receptable?  Are  plug-in  boards  keyed  to  prevent  improper 
ineertion? 

7-  Uectrical_Lnterferencc 

(1)  Do  all  the  provisions  of  specification  MIL-I-26t00  epply, 
or  should  some  waivers  be  sought? 

(2)  Vhet  tests  have  beer  performed  for  electricel  noise? 

(3)  Has  the  chassis  or  frame  been  grounded?  Have  shock  mounts 
been  bypassed  with  ground  streps?  Hes  the  insuleted  protective 
finiah  been  removed  where  a  metal-to-metel  contact  is  required? 

(4)  Are  openings  (such  as  those  for  access,  vantiletion,  and  cese- 
mounted  components)  shielded  to  prevent  cese  leakage?  Are 
eccess  doors  of  the  metel  1  extile  or  finger  atrip  type? 

(5)  Are  heeters  wired  with  twisted  or  isolated  leade? 
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Are  oscillators  isolated  from  other  ateges  and  from  antennas? 
Is  oscillator  power  kept  to  a  minimum?  Is  the  oscillator 
heater  decoupled  from  B  supply  sources? 

(7)  Do  paresitic  oscillations  exist,  end  is  suppression  necessary? 

(8)  Is  ur desired  signal  transfer  reduced  by  means  of  interstage 
decoupling  networks  and  link  or  parellel-tuned  circuits? 

(9)  JLre  pulse  networks  end  transformers  isolated?  Are  the  leeds 
associated  with  the  pulse  networks  decoupled?  Are  these 
leeds  kept  es  short  es  possible? 

(10)  Is  pulse  energy  fed  to  succeeding  stages  ir  cosxiel  leads 
where  possible?  (Guard  egainst  waveform  distortion  caused 
by  coaxiel  cable  capacitance.) 

(11)  Are  sharp  projections  avoided  in  high-voltege  circuits? 

(They  are  possible  sources  of  corona  and  arcing.) 

(12)  Are  sharp  bends  avoided  in  high-voltege  wiring?  (The 
possibility  of  insulation  breakdown  is  increased.) 

(13)  Are  the  magnetic  fields  associated  with  indicators  adequately 
isolated?  Are  indicator  control  and  power  leeds  decoupled 

by  the  use  of  feed-through  bypass  capacitors? 

(14)  Are  blower  motors  of  the  e-c  noncommutating  type? 

(15)  If  it  is  necessary  to  use  d-c  rotating  electrical  equipment, 
is  the  design  such  es  to  minimize  the  effects  of  the  commuta¬ 
tion  process?  To  this  end,  does  the  equipment  employ  such 
devices  ss  interpoles,  laminated  brushes,  es  large  a  number 
of  armature  coils  snd  commutator  bars  ss  possible,  and  good 
mechanical  design  and  construction? 

(16)  Is  relay  or  switch  operation  likely  to  create  power  supply 
transients  in  other  units  or  circuits? 

(17)  Has  consideration  been  given  to  arc  suppression  during  the 
making  or  breaking  of  switches  or  contacts?  (Several 
methods  are  available,  e.g. ,  e  simple  RC  network  across  the 
switch  or  contacts,  e  high  resistance  or  rectifier  across 
the  inductive  circuit,  negative  voltage  characteristics 
resistors.  If  these  are  inadequate,  shielding  snd  feed¬ 
through  capacitors  in  the  input  and  output  leeds  may  be 
required . ) 

(18)  Are  gas  tube  beater  supplies  and  output  leads  well  decoupled 
and  Isolated? 
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(19)  In  power  supplies  uaing  gas-tube  rectifiers,  is  use  made  of 
line  filters,  electrostatically  shielded  transformers,  and 
hash-suppression  chokes  in  tha  plate  and  cathode  leads? 

(20)  Are  electronically  regulated  power  supplies  provided  with 
decoupling  circuits  to  prevent  oscillations  in  the  regulator? 
Are  long  leads  avoided  in  the  plate  end  grid  circuits? 

EOT  H  -  MBCflAN  KM, 

1.  £gpe raj  Peffjgp 

(1)  Hes  use  of  cantilever  mounting  for  pmrts  end  assemblies  been 
minimised,  and,  where  used,  is  the  center  of  gravity  located 
near  the  mounting? 

(2)  Has  the  chassis  been  properly  designed  for  ite  application? 

(3)  What  are  the  locations  and  load  ratings  of  mounting  point e? 

(4)  Where  ere  the  heaviest  parte  located? 

(5)  Are  all  large  parts  and  assemblies  securely  mounted? 

(6)  Hes  the  center  of  gravity  been  considered  in  terms  of  the 
proper  distribution  of  shock  mounts? 

(7)  In  the  case  of  terminal  boards,  are  the  critical  components 
mounted  at  the  edges  rather  than  at  the  center,  and  are  they 
properly  supported? 

(8)  In  the  case  of  lead-mounted  parts,  have  weight,  lead  weight, 
thermal  expansion,  supplementary  support,  bend  rate,  and 
other  mounting  considerations  been  evaluated? 

(9)  Have  clearances  been  provided  with  dua  consideration  for 
vibration,  shock  and  noise  stresses? 

(10)  Can  electrical  inatebility  be  cauaed  by  vibration  of  mechanical 
parts? 

(11)  Have  shock  and  vibration  tests  been  performed?  If  not  ere 

they  scheduled?  9 

(12)  Hes  the  cooling  design  been  analyzed  to  provide  s  temperature 
contour? 

(13)  Are  heat  diaaipating  element  a  properly  located  with  reapect 
to  heat  aenaitive  perta?  Ia  there  aultable  flow  of  air? 
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(14)  Hava  component  parta,  subassemblies  and  assemblies  been 
supported  and  clamped  properly  with  adequate  consideration 
for  heat  dissipation? 

(15)  Is  the  unit  of  the  lightest  weight  consistent  with  sturdiness, 
safety  and  reliability? 

(16)  Are  all  items  visually  and  physically  accessible  when  the 
unit  is  on  the  test  stand? 

(17)  Is  the  possibility  of  physical  damage  to  the  unit  due  to  misuse 
of  adjustments  minimized  by  the  design? 

(13)  Is  the  possibility  of  damage  to  the  unit  during  handling  and 
installation  minimized  by  the  design7 

(19)  Can  the  unit  be  removed  and  replaced  within  the  required 
time  limit? 

(20)  Is  the  packaging  scheme  such  aa  not  to  impose  unrealistic 
spare  parts  requirements? 

(21)  Does  each  part  of  the  unit  designed  as  non-field  repairable 
meet  the  minimum  reliability  requirement  for  this  classifica¬ 
tion? 

(22)  Have  suitahle  heat  treatments  been  called  out? 

(23)  Has  design  heen  based  on  standard  tooling  wherever  possihle? 

(24)  Have  radii,  fillets,  curves,  and  straight  lines  been  suffi¬ 
cient  to  .give  all  possihle  freedom  to  manufacturing? 

(25)  Have  tha  most  economical  parts  satisfactory  for  the  applica¬ 
tion  been  specified  in  all  caaes? 

(26)  Are  all  purchased  components  called  out  hy  MIL,  AN  or  RCA 
(not  vendor)  numbers? 

(27)  Ara  the  components  arranged  and  mounted  for  the  most  econom¬ 
ical  assembly  and  wiring? 

(28)  Ara  all  fasteners  large  enough  for  their  application? 

(29)  Are  guide  pins,  keys  and  latches  of  sufficient  strength? 

(30)  Is  the  basic  structure  of  sufficient  strength  for  the 
application? 

(31)  Is  tha  design  such  as  to  prevent  excessive  radiation  into  or 
out  of  the  unit? 
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(32)  Are  parts  located  to  provide  for  logical  wiring? 

(33)  Ara  luhricetion  points  minimized?  Whera  required,  are  they 
accessible  and  clearly  marked? 

(34)  Is  the  predicted  reliability  within  the  unit  requirement? 

(35)  Have  unit  environmental  tests,  including  temperature  measure 
ments  at  key  points,  heen  completed?  If  not,  are  they  sched 
uled? 

(36)  Have  all  prohlems  highlighted  in  the  preliminary  design 
review  been  resolved? 

(37)  Has  there  been  compiled  a  separate  list  of  recommendations 
for  product  improvement  or  redesign? 

(38)  What  alternate  designs  were  considered? 

(39)  Have  the  appropriate  standards  been  consulted  for  materials, 
components,  drafting,  manufacturing  and  workmanship? 

(40)  What  factors  influenced  the  choice  of  this  particular  design 

(41)  Do  firm  specifications  exist,  including  test  specifications? 

(42)  Have  all  specifications  beer  mat  unconditionally? 

(43)  Does  any  specification  require  modification? 

(44)  Can  any  unreasonable  or  unusually  difficult  requirements  be 
relaxed? 

2.  Workmanship  and  Maintainability 

(1)  Is  soldering  adequately  specified?  What  provisions  have 
been  made  to  prevent  cold  joints  and  to  ensure  removal  of 
flux? 

(2)  Are  proper  screw  lengths  and  locking  provisions  specified? 

(3)  Are  designs  such  as  to  prevent  damage  to  components  durine 
installation? 

(4)  Have  guide  pins  been  provided  to  facilitate  installation  of 
plug-in  units? 

(5)  Are  plug-in  units  keyed  (by  some  means  other  than  the  connec 
tor )  to  prevent  accidental  insertion  in  the  wrong  location? 
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(6)  Hevs  tolerances  of  component  mounting  provisions  and  mating 
holes  been  coordinated? 

(7)  Have  sll  holes  been  located  for  enough  from  bends  to  prevent 
distortion? 

(8)  Are  bend  radii  specified  to  be  large  enough,  in  accordance 
with  appropriate  standards? 

(9)  In  reference  to  viring  and  cabling,  hsve  the  following  items 
bean  considered? 

a.  Doas  the  dasign  make  provision  for  properly  leading 
cables  around  cornars  and  sharp  edges? 

b.  Ara  grommets  provided  vhara  needed? 

c.  Is  tha  design  such  ss  to  minimize  soldaring  iron  bums 
during  both  manufacture  and  maintenance? 

d.  Is  lacing  properly  and  adequately  specified? 

a.  Have  harnesses  bean  properly  routed  and  has  sufficient 
clamping  bean  provided  to  prevent  cables  hanging  loose? 

f.  Has  adequata  space  bean  allowed  for  harnesses  and  for 
breakouts  to  connectors,  etc? 

g.  Ara  haavy  wires  being  brought  to  terminals  of  adequate 
size? 

h.  Are  stranded  wires  properly  secursd  close  to  soldar 
joints  to  prsvent  flexing? 

i.  Is  any  cable  (or  wire)  overly  taut,  with  strain  being 
placed  on  tha  connector  (or  connection),  tha  cable  (or 
wire)  or  tha  clamps? 

j .  Do  any  cablas  or  wires  lia  across  removable  units  or 
across  fasteners  of  any  type? 

k.  Ara  all  connectors  visibla,  and  sra  they  aasily  accessible 
to  tools  and  hands? 

l.  Hava  cablas  (wiras)  and  connactors  (connections)  been 
properly  identified?  Can  wrong  connections  rasult  from 
cable  layout  and  connector  type? 

m.  Do  any  cabla  (wire)  runs  permit  contact  betwean  the  cable 
(wirs)  and  moving  perta? 
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(10)  Are  ell  items  (parts  and  subassemblies)  visually  snd  physically 
accessible  for  assembly,  wiring  rework  snd  maintenance? 

(11 )  Are  all  test  points  sccassibla  when  tha  unit  is  properly 
installed? 


(12)  Ara  all  field  sdjustmenta  accessible  when  the  unit  i3  properly 
installed? 

(13)  Has  sequent Isl  assembly  been  svoided  which  results  in  involved 
sequential  disassembly  in  order  to  make  repairs  and  ad justmants? 

(>4)  Is  the  design  such  that  no  unrealistic  requirements  for  special 
facilities  for  maintensnca,  storage  or  shipment  sra  imposed? 

05)  Is  the  design  such  that  no  unnecessary  requirements  for  e 
special  maintenance  environment  (s.g.,  ground  power  carts, 
cooling,  special  primary  power,  ate.)  ara  imposed? 

(16)  Does  the  dasign  provide  for  adaquata  protaction  of  maintenance 
snd  test  personnel  against  accidental  injury? 

Materials  srd  Processes 

(1)  Hsve  stsndsrd  msterisls  been  specified  in  all  possible  cases? 

(2)  Have  the  most  economical  matsrisls  and  processes  suitable 
for  the  applications  bean  specified  in  all  cases?  (Material 
cost,  fabrication  cost  snd  finishing  cost  should  be  considered.) 

(3)  Hava  corrosion-resistant  materials  or  finiahas  been  provided? 

(4)  Are  there  dissimilar  metals  in  contact? 

(5)  Are  sll  materials  satisfactory  for  the  temperatures  involved? 

(6)  Is  the  possibility  of  flaking  considered? 

(7)  Hss  moisture  protection  bean  provided  where  necessary? 

(8)  Ara  all  materials  fungus  resistant  or  inert? 

(■9)  Are  electrically  conductive  finishes  provided  where  necsssary? 

(10)  Hsve  machine  finishes  bean  reviewed  for  the  most  economical 
processes  suitable  for  the  requirements? 

(11)  Have  rivet*  or  spot  wslds  been  specified  where  possible  in 
preference,  to  welding,  furnace  brazing,  .etc? 
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(12)  Has  each  sheet  metal  piece  been  examined  to  determine  whether 
it  has  too  many  bends  for  economical  fabrication? 

PART  III  -  HUMAN  WQim&m 

(1)  Are  visual  indicators  mounted  so  that  operator  can  see  scales, 
indices,  pointers  or  numbers  clearly?  Are  scale  graduations, 
design  of  numerals  and  pointers,  and  scale  progressions  presented 
so  that  accurate  reading  is  enhanced? 

(2)  Do  visual  displays  have  adequate  means  for  identifying  an  opera¬ 
tive  condition? 

(3)  Have  amhiguous  information  and  complicated  interpolations  been 
eliminated  from  visual  indicators  to  minimize  reading  errors? 

(4)  Do  controls  work  according  to  the  expectation  of  the  operator? 
(Naturalness  of  movement  direction  is  derived  from  previous 
experience  as  well  as  certain  handedness  factors.) 

(5)  Do  functionally  related  controls  and  displays  maintain  functional 
or  physical  compatibility,  such  as  direction-of -motion  relation¬ 
ships  or  proximity  to  each  other? 

(6)  Are  controls  designed  so  that  the  operator  can  get  an  adequate 
grip  for  turning,  twisting  or  pushing? 

(7)  Does  console  design  provide  knee  room,  optimum  writing  surface, 
height,  or  optimum  positions  for  controls  and  displays? 

(8)  Do  equipment  design  and  arrangement  allow  space  for  several 
operators  to  work  without  interfering  with  each  other? 

(9)  Do  arrangement  and  layouts  stress  the  importance  of  halancing 
the  workload,  or  do  they  force  one  hand  to  perform  too  many 
tasks  while  the  other  hand  is  idl£? 

(10)  Is  the  illumination  designed  with  the  specific  task  in  mind, 
rather  than  with  a  general  situation?  (Many  instruments  are 
practically  useless  because  of  lack  of  illumination.) 

(11)  Have  extreme  glare  hazards  been  eliminated,  such  as:  brightly 
polished  bezels,  glossy  enamel  finishes,  or  highly  reflective 
instrument  covers? 

(12)  Are  assemblies  and  parts  stacked  so  that  some  have  to  be  removed 
to  repair  or  replace  others,  thus  complicating  maintenance? 

(13)  Do  fasteners  for  chassis  and  panels  require  special  tools  which 
hamper  maintenance? 
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(14)  Do  chassis  door  slides  have  means  for  holding  the  unit  extended 
for  servicing?  Are  the  slides  too  loose,  or  do  they  hind? 

(15)  Are  handles  provided,  and  are  the  chassis  or  units  light  enough 
to  be  moved  without  undue  strain? 

(16)  Is  calibration  indexing  provided  for  maintenance  adjustment  and 
calibration  adjustment  controls?  (Screwdriver  adjustments  are 
often  too  sensitive.) 

(17)  Do  the  coding  and  symbols  on  ecuipr.eMs  end  in  instruction 
manuals  coincide'  (loo  ^ew  books  1ojl  what  or  how  to  check, 
what  to  expect,  or  how  lo  correct ,  red  when  covered,  the  informa¬ 
tion  is  not  organized  so  that  it  may  be  found  quickly.) 

(18)  Is  illumination  provided  for  the  maintenance  technician? 

PART  IV  -  VALUE  ENGINEERING 

1 .  Specification  Review 

(1)  Have  the  customer's  specifications  heen  critically  examined 
to  see  whether  they  ssk  for  more  than  is  needed? 

(2)  Has  the  cost  of  any  overdesign  been  defined  for  its  effect 
on  production  as  well  as  on  the  R&D  program? 

(3)  Has  the  cost  effect  of  contract-required  overdesign  been 
discussed  with  the  customer? 

2.  General 

(1)  Does  the  design  give  the  customer  what  he  requires  and  no 
more? 

(2)  Could  costs  be  radically  reduced  hy  a  reduction  of  performance, 
reliability,  and/or  maintainability  to  the  minimum  specified? 

(3)  Could  costs  be  radically  reduced  hy  a  reduction  of  resistance 
to  high  temperature,  shock,  vibration  or  other  environments 
to  the  minimum  specified? 

(4)  Have  circumstances  changed  (changes  in  concept  or  specifica¬ 
tion,  progress  in  the  art,  development  of  new  components  or 
processes)  so  that  the  design  includes  unnecessary  or 
expensive  circuitry,  parts  or  processes? 

(5)  Have  unnecessarily  high  cost  items  been  included  as  a  result 
of  their  availahillty  when  the  breadboard  or  model  was 
constructed  ? 
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(6)  Cen  any  varisble  devices  euch  ee  potentiometers  included 
for  breedboard  or  model  operetionel  edjuetmente  be  chenged 
now  to  fixed  component  part  *  or  semied justeble  designs? 

3.  Production  Coats 

(1)  Ara  the  quantities  to  be  built  on  this  order  known?  Are 

the  estimated  quantities  to  be  built  on  future  orders  known? 

Hava  thesa  factors  been  considered  in  the  design  decisions? 

(2)  Will  tooling  costs  be  in  lina  with  present  end  anticipated 
production? 

(3)  How  much  do  you  estimate  the  design  will  cost  in  production? 

4.  Electronic  Design 

(1)  Does  the  design  represent  optimum  electricsl  simplicity? 

(2)  Is  circuitry  overly  complex  or  conservative? 

(3)  Heve  etandard  "preferred  circuits"  been  reviewed  to  see  how 
many  can  be  used  beneficielly? 

(4)  Hse  tha  field  of  commercielly  available  packaged  circuits, 
power  supplies,  etc.  been  reviewed  against  your  requirements? 

(5)  Can  circuitry  be  alimineted  by  having  one  circuit  do  the  job 
of  two  or  more? 

(6)  When  specifying  special  component  parts,  have  potential 
vendors  been  consulted  for  elternatives  or  modifications 
that  would  hold  costs  down? 

(7)  Hsve  ell  high  cost  components  such  as  transistors,  semicon¬ 
ductor  diodaa,  magnetic  and  high  power  davicas,  motors,  gasr 
treins  and  decoders  been  examined  to  determine  whether  lower 
cost  eubstitutions  cen  be  made? 

(8)  Ara  the  components  the  lowest  cost  meeting  the  design  require¬ 
ments? 

(9)  Can  any  alactrical  tolerance  be  liberalized  to  ellow  specifica¬ 
tion  of  lower  cost  parts? 

(10)  Have  nearly  identical  parte  been  made  identical  to  gain  the 
edventage  of  quantity  buying  or  manufacture? 

(11)  Has  coax  ceble  bean  epecified  when  hookup  wire  or  shieldad 
ceble  will  do  the  job? 
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(12)  Hes  silicon  been  specified  for  traneietors  or  diodae  when 
germanium  will  do  the  job? 

(13)  Can  metalized  Mylsr  be  substituted  for  tanteium  or  Cerafil 
capacitore? 

(14)  Have  automated  techniques  bean  used  to  tha  maximum? 

(15)  Is  Teflon  wire  specified  whera  other  insulation  will  suffice? 

5.  Mechanical  Design 

(1)  Does  the  design  represent  optimum  mechanicel  simplicity? 

(2)  Is  every  part  ebsolutely  necessery?  Csn  any  part  be  eliminated 
or  combined  with  another  port  to  reduce  total  number  of  parte 
and  cost? 

(3)  When  specifying  special  parts,  have  potential  vendors  been 
consulted  for  elternatives  or  modifications  that  would  hold 
costs  down? 

(4)  Are  machsnicsl  tolerances  within  the  limits  of  normal  shop 
practice  defined  in  RCA  Spec.  96400?  Csn  any  tighter  tolerance 
called  out  be  changed  to  agree  with  RCA  Spec.  96400,  or  be 
liberalized  to  hold  costs  down? 

(5)  Are  the  surface  finishes  tha  coarsest  that  will  do  the  job? 

(6)  Are  tha  fabrication  processes  the  lowest  cost  meeting  the 
design  requiremente? 

(7)  Have  nearly  identical  parts  been  mada  identical  to  gsin  the 
advantage  of  qusntity  buying  or  manufacture? 

(8)  Are  tha  materials  the  lowest  cost  meeting  the  design  require¬ 
ments? 

(9)  Does  the  combination  of  material  snd  protective  finish 
specified  result  in  the  lowest  cost  combination? 

(10)  Has  cognizance  been  taken  of  relative  workability  of  materials? 

(11)  Have  stendsrd  elloys,  grades  and  sizes  of  stock  been  specified 
whenever  possible? 

(12)  Csn  the  design  be  altered  in  eny  raepect  to  evoid  the  usa  of 
non-standard  tooling?  Saa  RCA  drefting  stendard  8-224-200 
series. 
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(13)  Hes  the  l/lO"  grid  drafting  system  for  sheet  metal  parts 
been  used  wharaver  epplicahle? 

(14)  Can  the  design  be  modified  to  enable  tha  use  of  the  same 
tooling  for  right  and  left  hand  or  similar  parte? 

(15)  Ara  drawings  for  fabrication  of  parts  which  are  similar  to 
parts  already  produced  cross  referenced  so  availahle  tooling 
can  be  used? 

(16)  Cen  the  design  be  altered  to  avoid  unnecessary  handling  and 
processing  resulting  from  such  things  as  riveting  and  spot 
walding  on  the  eame  suhassemhly  part? 

(17)  Hava  automated  techniques  been  used  to  the  maximum? 

(18)  Are  casting  bosses  of  adequate  si2e,  considering  the  large 
tolerances  which  epply  to  casting  dimensions? 

(19)  Can  cores  or  complex  parting  lines  be  eliminated  from  any 
casting  hy  moderate  redesign? 

(23)  Is  impregnation  of  castings  called  out  when  it  would  aid 

processing?  (Castings  should  he  impregnated  ffter  machining 
if  they  are  to  be  electroplated.  This  impregnation  prevents 
absorption  of  plating  acids  or  salts.  Castings  should  also 
be  impregnated  if  they  are  to  hold  liquids  or  gases  under 
pressure . ) 

(21)  Have  engineering  and  factory  specialists  been  consulted  for 
castings,  forgings,  weldments,  heat  treatment  and  other 

specialties? 

(22)  Have  standard  sizes,  gredes  and  alloys  of  fasteners  been 
specified  whenever  possible? 

(23)  Are  all  manual  welding  operations  specified  absolutely 
necessary?  Can  furnace  hrazing  be  substituted? 

(24)  Are  the  assemhly  processes  the  lowest  cost  meeting  the 
design  requirements? 

(25)  Has  adequate  clearance  between  parts  been  provided  to  allow 
for  easy  assamhly?  (Parts  have  hecome  smaller  hut  hands 
have  not . ) 

(26)  Are  all  parts  designed  for  assembly  at  the  earliest  possihle 
time?  Assemhly  costs  go  up  ae  the  huildup  of  the  system 
progresses . 
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(27)  Are  markings  adequate  to  guide  the  assemhly  processes? 

(28)  Hava  the  engineering  and  factory  specialiate  been  consulted 
on  any  unusual  assemhly  prohleras ? 

(29)  Has  datum  line  rather  than  multiple  surfeca  dimensioning 
been  used  on  all  drawings? 

(30)  Can  any  four-placa  dimension  he  changed  to  a  threa-plece 
dimension? 

(31)  Can  any  three-place  dimension  be  changed  to  a  two-place 
dimension? 

(32)  Can  heat  treating  after  forming  sheet  metal  parts  be  elimi¬ 
nated  hy  change  of  design  or  material  to  avoid  straightening 
prohlams  ? 

(33)  Is  all  masking  from  finishing  materials  (such  as  plating 
solutions  and  paint)  necessary? 

6.  Standardization 

(1 )  Have  you  coordinated  your  design  with  those  who  may  be  using 
similar  (or  have  used  in  the  past)  designs,  circuits,  parts 
or  components  to  get  optimum  benefit  from  standardization 
and  past  experience? 

(2)  Are  the  standerd  circuits,  standard  components  and  standard 
hardware  the  lowest  cost  standards  which  will  supply  the  min¬ 
imum  required  characteristics? 

(3)  Can  the  use  of  each  nonstandard  part  or  circuit  he  adequately 
justified  ? 

(4)  Can  any  new  nonstandard  part  be  replaced  by  a  nonstendard 
part  which  hes  elready  been  ECA  E-Form  approved? 

(5)  Do  control  drawings  leave  no  question  that  a  vendor  standard 
part  is  being  specified  when  such  is  intended? 

(6)  Has  standardization  been  carried  too  far  until  tha  cost  of 
excess  function  is  greater  than  the  gains  resulting  from 
high  quantity? 

7.  Maintainability  Design 

(1)  Is  each  assembly  self-supporting  in  the  desirahla  position 
or  positions  for  easy  maintenance? 
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(2)  Can  assemblies  be  laid  on  n  bench  in  any  position  without 
damaging  components? 

8.  Testing 

(1)  Are  the  teet  proceeeee  the  loveet  coet  meeting  the  design 
requirements? 

(2)  Can  any  test  epecification  be  eliminated  or  relaxed? 

(3)  Have  interacting  controls  been  eliminated  or  the  adjustments 
specified  in  euch  a  manner  that  the  lowest  cost  factory  tsst 
personnel  can  easily  align  the  circuit? 

(4)  Ie  the  system  competibls  with  the  requirements  for  checkout 
in  the  factory  -  if  not  as  a  complete  eystem,  then  in  large 
subsystem  segments? 

(5)  Have  the  teet  procese  experts  beer,  consulted  for  alternatives 
that  would  keep  their  costs  down? 

9.  Subcontract  Items 

(1)  Has  the  field  of  commercially  available  packaged  units,  sub- 
assemblise  and  circuits  been  thoroughly  reviewed  to  be  sure 
there  are  no  standard  vendor  items  that  will  do  the  job? 

(2)  Is  desired  cost  control  adequately  emphasized  in  subcontract 
specifications? 

(3)  Have  our  specifications  for  subcontract  items  been  reviewed 
against  the  check  list  to  be  sure  we  ars  not  overspecifying? 

(4)  Have  suggestions  been  invited  from  prospective  suppliers  re- 
gsrding  possible  value  improvement  from  loosening  specifica¬ 
tion  limitations? 
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SUMMARY  OF  CIRCUIT  TECHNIQUES 
FOR  RELIABILITY  DESIGN  ANALYSIS 

Just  ae  "the  kingdom  was  lost  for  want  of  a  nail",  so  is  it  poseible 
to  render  inoperable  an  sxpeneive,  complex  command  and  oontrol  system 
for  want  of  a  reliable  circuit. 

It  ie  poor  design  strategy  to  plan  to  apply  highly  reliable  com¬ 
ponent  parts  (euch  as  the  well  advertised  MINUTEMAN  parts)  in  marginally- 
designed  circuitry  and  expect  effective  circuit  performance.  An  impor¬ 
tant  phaes  of  an  equipment  subsystem  design  rsview  is  an  examination  of 
circuit  deeign  for  reliability. 

Circuit  analysis  techniquee  currently  available  range  from  a  brief 
review  of  voltagee,  currents,  and  power  strssese  that  a  circuit  would 
be  subjected  to,  either  during  normal  operation  or  at  woret-case  con¬ 
ditions,  with  hand  computations,  to  computer-mechanized  techniques  for 
handling  more  complex  circuitry  and  providing  a  more  detailed  analysis. 

Computer  orientated  circuit  analysis  attempts  to  simulate  a  circuit 
mathematically  on  a  computer  and  show  how  the  performance  of  a  circuit 
will  behave  as  its  basic  component  parts  deteriorate  during  life.  This 
is  accomplished  by  programming  circuit  equations  into  a  computer  and 
methodically  varying  the  values  of  ths  circuit's  part  parameters. 

Very  briefly,  computer  methods  of  circuit  analysis  entail  the  follow¬ 
ing  steps: 

a.  The  drawing  of  an  equivalent  circuit. 

b.  The  writing  of  equivalent  circuit  equations  and  circuit 
requirements  in  terms  of  part  parameters  and  reducing  to  matrix  form. 

c.  Incorporating  the  equations,  circuit  requirements,  and  ■ 
desired  part  parameter  variation  changes  into  a  computer  program. 

d.  Debugging  and  running  the  computer  program. 

e.  Plotting  and  analyzing  the  computer  output. 

At  least  five  well  defined  computer -mechanized  methods  are  presently 
operational.  Each  method  is  considerably  different  from  the  other. 
Although  it  might  appear  that  one  msthod  should  be  sufficient,  there  are 
two  basie  reasons  for  the  multi type  analyses: 

a.  Since  there  are  many  types  of  circuits  in  a  command  and 
control,  weapon,  or  support  system,  any  one  method  would  not  be  the  best 
method. 
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b.  As  detailed  data  on  parte  parameters  became  available, 
statistical  methods  of  circuit  analysis  were  prepared  to  take  advantage 
of  the  new  information. 

Of  the  methods  available,  the  Parameter  Variation  method  requires 
the  least  amount  of  input  data.  The  only  necessary  input  data  are  the 
upper  and  lower  limits  of  each  input  parameter.  Because  of  their 
general  statistical  data,  the  Moment  and  Monte  Carlo  methods  require 
the  most  extensive  input  data  of  the  various  methods.  The  Moment 
method  requires  the  mean  value  and  variance  of  each  parameter,  while 
the  Monte  Carlo  method  requires  the  entire  frequency  distribution  of 
the  parameter.  These  limitations  may  prevent  the  two  methods  from  being 
widely  applied. 

The  Mandex  Worst-Case  Method  is  based  on  a  design  philosophy  which 
is  relatively  easy  to  comprehend;  i.e.,  if  a  circuit  will  function 
properly  with  its  parts  at  their  worst-case  condition,  the  circuit  should 
operate  with  any  combination  of  part  characteristics  as  long  as  their 
worst-case  condition  is  not  exceeded. 

The  Mandex  Worst-Case  analysis  has  expanded  this  general  philosophy 
on  the  assumption  that  there  is  not  just  one  possible  worst-case  condi¬ 
tion  for  a  circuit,  but  one  for  each  output  variable;  therefore,  a  com¬ 
plete  worst-case  analysis  is  performed  on  each  output  variable. 

The  name  VINIL  was  obtained  from  the  nature  of  this  method  of  analy¬ 
sis  where  Vj^  is  swept  from  its  minimum  to  maximum  end  of  life  value, 
and  the  output  parameter  of  interest  (^l)  is  plotted  for  each  sweep 
increment.  The  analysis  of  the  results  of  the  VINIL  method  is  straight¬ 
forward  in  that  the  graphs  are  simple  input-parame ter  versus  output- 
parameter  graphs. 

Returning  briefly  to  the  Moment  Method,  this  method  of  circuit  analy¬ 
sis  is  based  on  the  well  known  theorem  on  the  Propagation  of  Variance 
which,  in  part,  states: 

(T‘t  =  (h[  )2  of,  -r  (h!,)2  of  +  za[  h]  Ci  Oj  + . 

where 

2 

CT  =  the  output  parameter  variance 
2 

O’ ^  =  the  variance  of  input  parameter  n 

H1  =  the  partial  of  the  output  parameter  with  respect 
to  input  parameter  n 
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a^d  fjj  =  the  correlation  coefficient  of  parameters  i  and  j 


For  normal  distributions,  95  percent  of  the  samples  will  fall  within 
_  (T  about  the  mean  value.  For  abnormal  distributions,  according  to  the 
Tchebycheff  Theorem,  +  4. 5  0" must  be  used  to  be  95  percent  inclusive. 
Knowing  these  facts,  it  becomes  possible  to  place  restrictions  on  the 
mean  output  value  by  stipulating  that  it  must  be  at  least  ACT  from  some 
value . 


Attachment  2 
Page  3  of  3 


U5 


t 


SECTION  IV 

GENERAL  REQUIREMENTS  FOR  A 
RELIABILITY  AND  MAINTAINABILITY 
DATA  COLLECTION  AND  EVALUATION  SYSTEM 
FOR  ELECTRONIC  SYSTEMS 


% 


- 


SECTION  IV 


GENERAL  REQUIREMENTS  FOR  A  RELIABILITY  AND  MAINTAINABILITY 
DATA  COLLECTION  AND  EVALUATION  SYSTEM  FOR  ELECTRONIC  SYSTEMS 


FOREWORD 


The  purpose  of  tnis  section  is  to  provide  guidance  to  SPO  Reliability  and 
and  Maintainability  (R/m)  Monitors  in  establishing  requirements  for  a  contractor  s 
data  collection  and  evaluation  system. 

Section  2  presents  general  requirements  for  an  R/M  Program  Plan.  Briefly 
discussed  therein  is  the  task  of  data  collection  and  evaluation.  This  section 
will  discuss  the  task  in  greater  detai-l.  Data  collection  and  evaluation  provides 
the  necessary  technical  and  management  information  associated  with  the  development, 
checkout,  and  delivery  of  electronic  systems  and  equipment. 
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GENERAL  REQUIREMENTS  FOR  A  RELIABILITY 
AND  MAINTAINABILITY  DATA  COLLECTION  AND  EVALUATION 
SYSTEM  FOR  ELECTRONIC  SYSTEMS 


1 •  Jptr94uctlon 

It  is  a  we  11 -jcnown  fact  that  a  certain  amount  of  foreknowledge  must  be 
possessed  by  a  SPO  of  a  problem  before  e  proper  decision  can  be  made  for  ita 
elimination.  The  same  holds  true  within  e  contractor's  organization.  His 
engineering,  manufacturing,  and  management  personnel  muet  continuously  assess 
the  equipment  during  ell  phases  of  daaign,  development,  production,  and  field 
use.  With  an  accurate  and  complete  knowledge  of  the  problems  end  ahort- 
comings  concerning  his  equipment,  it  is  most  probable  that  the  contractor's 
recommendations  for  changee  in  design  and  procedures  will  et  laajt  be  of 
sound  foundation.  It  ia  also  Important  to  know  that  there  ara  no  problems, 
if  this  be  the  case. 

This  pamphlet  presents  the  basic  methods  and  ressons  for  contractors  to 
design  and  implement  a  data  collection  and  evaluation  eyatam  es  part  of  their 
overall  planning.  It  is  desired  that  the  R/fo  Monitor  will  have  a  basic 
knowledge  of  tha  salient  ingredients  of  a  data  collection  eystem  for  usa 
during  each  phase  of  the  overall  program  to  the  and  that  he  can  specify  e 
syatem  best  suited  for  the  needa  of  the  overell  SPO  mission.  Further,  ha  can 
more  properly  essesa  the  contractor's  proposed  plan  end  subsequent  progress 
to  assure  that  tha  contractor  has  implemented  his  plan  to  be  useful  and 
effective . 

2.  Time-Phased  Failure  and  Repair  Data  Collections  Methods  and  Applications: 

It  must  be  kept  in  mind  that  the  task  of  data  collection  in  itself  is  not 
to  tha  and  that  the  deta  terminates  in  e  fils  or  tha  wastebasket;  the  letter, 
most  likely.  Deta  collection  is  only  one  task  of  several  in  e  supposedly  well- 
conceived  contractor's  R/fo  Program  Plan.  Developed  hare  are  the  beaic  requira- 
menta  of  a  deta  collection  syatem,  data  aourcas,  and  deta  usas.  Thie  pamphlet 
is  concerned  with  three  major  phases  of  cn  overall  program;  namely,  ths  design 
end  development  phese,  the  manufacturing  cr  production  phase,  and  ths  opera¬ 
tional  or  field  evaluation  phaee. 

3.  Design  and  Development  Phage  Requirements.  This  phase  of  a  program  is 
moat  important  in  thet  it  is  during  this  phaaa  that  system  inherent  availa¬ 
bility  is  plenned  end  established. 

a.  Data  Sources.  It  is  not  unusual  to  expect  to  find  various  test  programs 
being  conducted  et  this  time.  Examples  of  these  teats  era  those  conducted  st 
the  part  level,  breedboard  and  prototype  esasmbly  end  aubeaasmbly  levels,  and 
many  times,  et  the  prototype  system  lsvsl. 
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Some  very  meaningful  R/M  data  result  fTom  initial  teats  performed  in 
the  engineering  laboratory  under  either  room  ambient  or  controlled  environmental 
conditions.  The  contractor's  test  engineers  usually  maintain  notebooks  which 
contain  entries  of  test  conditions,  elapsed  time  indicator  readings,  and  teat 
results.  It  is  expected  that  the  contractor’s  collection  system  provides  for 
the  routine  collection  of  these  data,  either  by  completion  of  failure  report 
forma,  by  test  personnel,  or  by  lifting  the  deaired  data  from  the  teat  logs  by 
the  contractor’s  R/M  personnel,  or  a  combination  of  both.  It  is  very  important, 
during  this  early  planning,  thet  due  consideration  be  given  to  the  total, 
planned  test  program  -  not  only  thoae  teats  that  are  to  be  performed  daring 
the  design  and  development  phase,  but  for  all  phasea  of  the  overall  program  as 
sources  for  R/M  data-  It  is  at  the  beginning  of  a  proposed  program  that  the 
contractor’s  R/M  engineers  'should  plan  and  coordinate  with  other  activities  for 
their  total  data  needs  acd  the  nammer  in  which  these  data  will  be  time-phased 
as  inputs  for  use  during  the  performance  of  the  other  R/4l  tasks. 

b.  Nature  of  Data.  When  considering  the  kinds  of  data  needed,  thought 
must  be  given  to  its  subsequent  use. 

The  first  and  foremost  reason  for  the  contractor  to  collect  accurate 
failure  ana  repair  cata  is  to  evaluate  it,  and  detect  and  correct  design  and 
procedural  problems  as  early  as  possible  in  the  formulation  of  the  deaign. 
Subsequently,  the  data  provide  necessary  feedback  for  the  determination  of 
corrective  action  effectiveness.  It  also  provides  early  inputs  to  the  verifica¬ 
tion  or  modification  of  part  failure  rates  and  feasibility  of  circuitry  deaign. 
Some  of  the  deaired  information  is  aa  followa: 

(i)  Equipment  identifications. 

(*)  Test  conditions  and  environments  (bench  test  room  ambient, 

RFI,  vibration,  etc.). 

(3)  Elepsed  time  indicator  readings  (standby  end  operate). 

(4)  Date. 

(5)  Teat  rasulta. 

If  a  malfunction  occurs,  additional  information  ia  naaded  to  properly  eveluate 
the  cause;  thia  information  includear 

(fc)  Failure  aymptoma. 

(7)  Elapsed  time  indicator  resdinga  at  time  of  failure. 

(8)  Corrective  maintenanca  action  to  reatora  operation. 
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l°<  £“1  reliability  data  haa  mainly  been  considered.  Of  concern,  also  is  the 
kind  of  data  in  support  of  meintainability  studies,  and  includaa: 

(9)  Time  to  locate  trouble. 

(10)  Time  to  remove  and  replece  faulty  elements. 

(11)  System  checkout  time. 

(12)  Time  to  repair  faulty  element. 

(13)  Preventive  maintenance  time. 

During  the  evaluations  of  the  contractor’s  plans  the  R M  Monitor  will 
Ztl  the  Plans  COntain  prOVl3lons  the  collection  of  the  above ^ 

„  ^iS-|2fila®tica.  Earlier  in  this  section  it  was  stated  that  the  prime 

collecting  failure  and  repair  data  was  to  detect  and  correct  design 
and  procedural  problems.  To  give  an  indication  of  the  worth  of  these  data  ^ 
the  typical  evaluation  flow  will  be  discussed.  Once  the  data  are  collected 

?akerol  a  tue  COn“r-tor's  R/M  er°uP’  the  following  steps  should  then 

take  place  on  each  completed  data  form 

velid-v(1ntnw1eng;neirS  scree\data  «ntriea  for  completeness  and  technical 
/tav  1  drawines >  Par -  purchase  specifications,  and  test  procedures. 

Any  incomplete  or  wrong  entries  are  rectified  as  aoon  as  possible . 

If  8  c°nt“=tor*s  plan  calls  for  coding  of  raw  data,  the  engineer 
indicates  proper  code  for  technical  data.  This  includes  the  coding  for 

o? f£!J?  (ZrTrTSZ  Tratl0n  ( inoperative ,  intermittent,  etc  ),  cause 
of  failure  (part  failure,  test  error,  incorrect  fabrication,  etc  )  and 
responsibility  (Quality  Control,  Design  Engineering,  User,  etc.). 

(3)  Data  reports  are  put  in  final  code  form  for  keypunching  by  data 
Snure  dfta^”  ReSUlta  °f  ^  ^e  integrald  wit^ 

MTBF  nm  .:ab/:itl°nVf  are  evaluEtad  by  statisticians  for 

f  trends,  and  by  engineers  for  week  links  and  design 

nf  df  y  alS0  ^  retrieTad  io  requested  sequence  in  support 

UoTsZ/ToZZ  °f  giV6n  applL- 
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(5)  Results  of  studies  end  evaluations  by  the  R/H  engineers  are  fed 
beck  to  the  eppropriate  ectivity  in  the  corrective  ection  feedback  loop  - 
design  problems  to  the  design  engineering,  manufacturing  problems  to  the 
Quality  Control  ectivity,  vendor  problems  to  the  procurement  ectivity. 

(6)  Results  of  corrective  actions  effectuated  are  fed  back  to  the 
R/H  activity  and  are  used  to  up-date  mathematical  models  and  predictions  of 
MTB7  and  MTTR. 

4.  Manufacturing  or  Production  Fhesa  Requirements - 

a.  Data  Sources.  To  fully  understand  the  worth  of  production  R/fa  deta, 
it  is  necessery  to  first  consider  the  besic  concepts  of  inherent  R/fa  and 
operetional  R/fo.  Inherent  R /H  may  be  defined  as  the  reliability  and  main¬ 
tainability  potential  present  in  the  design;  i.e.,  that  which  is  designed  in. 

This  R/H  potential  may  be  echieved  et  the  site  -  operationally  -  if  there  is 
no  degradation  due  to  fabrication  and  essambly.  Operational  R/H  is  the  R/H 
demonstrated  in  e  service  application.  In  th©  case  of  reliability,  it  consists 
of  inherent  reliability  degreded  by  manufacture,  test,  shipping,  handling, 
storage,  maintenance,  and  use.  Practically  the  same  may  be  said  about  the 
degradation  of  inherent  maintainability. 

As  sources  of  data,  than,  the  SPO  R/fa  Monitor  will  look  to  the  areas 
and  agencies  responsible  for  the  degradation  of  inherent  R/M ;  namely,  manufactur¬ 
ing  (production),  handling,  storage,  maintenance,  and  teat. 

b.  Nature  of  Data.  Data  csn  be  separated  into  broad  categories  as  quality 
data  and  R/H  data. Quality  data  includes  records  of  inspection  and  testing; 
e.g.,  go-no-go  teats,  measurements  of  variables  such  as  resistance  and 
capacitance  to  determine  conformance  to  established  technical  requirements 
contained  in  apecificetiona ,  drawings,  and  purchase  orders.  R/H  data  on  equip¬ 
ments  are  developed  during  preproduction  stages  in  order  to  detect  equipment 
weaknesses  before  releese  to  production  and  to  obtain  s  quantitative  estimate 
of  equipment  R/fa.  R/H  data  on  parte  and/or  component©  are  developed  during 
the  prodnction  stages  to  eaaura  that  the  equipment  inherent  R/fa  is  not  unduly 
degraded  by  manufacturing  procaaaea.  When  the  data  indicates  excessive 
failure  rates  or  excessive  times  to  repair,  this  information  must  lead  to 
corrective  ections. 

In  designing  a  failure  and  repair  form,  the  following  minimum  items 
will  be  considered  for  inclusion: 

(1)  Report  aerial  number. 

(2)  Report  activity. 

(3)  Reported  by  (name  or  individual). 
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(4)  Date  of  report. 

(5)  Dete  of  malfunction. 

(6)  Geographical  place  of  malfunction. 

(7)  Reference  or  circuit  designations. 

(8)  Name  of  item  that  malfunctioned. 

(9)  Nama  of  manufecturer  of  the  item. 

(10)  Manufacturer's  part  number. 

(11)  The  item's  serial  number. 

(12)  Symptom  of  malfunction. 

(13)  Type  of  malfunction. 

(14)  Environment  at  the  time  of  malfunction. 

(15)  Elapsed  time  to  failure  or  malfunction. 

(16)  Clock  time  to  isolate  feihure  csuse. 

(17)  Clock  time  to  actually  repair. 

(18)  Clocfk  time  equipment  wss  down  for  repair. 

(19)  Action  taken  to  clear  malfunction. 

t20)  Requirement  that  part  be  forwarded  to  reliability  organization 
for  ene lysis. 

(21)  Recommendations  and/or  comments. 

Paragraph  3. 5. “12  of  MIL-R-27542A  should  be  referred  to  for  a  more  complete 
listing. 


c*  Dsts  Evaluation.  A  prime  reeson  for  accumulating  and  analyzing  failure 
and  repair  data  during  the  production  phase  ia  to  evaluate  the  R/H  being 
achieved  during  the  fabrication  stages.  Date  releting  to  failure  end  repair 
should  be  forwarded  to  the  contractor 's  R/>4  groups.  It  should  be  the 
responsibility  of  this  gronp  to  evaluate  the  need  for  further  analysis  to 
determine  the  cause  of  feilura  or  to  make  this  determination  by  utilizing 
existing  data.  Figure  1  is  e  typicel,  basic  R/H  information  flow  between  e 
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contractor's  activities.  It  also  indicates  inputs  from  the  field,  end  feed¬ 
back  to  the  SPO.  Referring  to  this  figure,  it  ia  obvious  that  data  are  not 
collected  for  the  sake  of  collect ing  data .  It  should  be  noted  that  it  is  the 
responsibility  of  the  R/M  group  to  disseminate  to  the  various  interested 
groups  information  which  clearly  indicates  the  need  for  improvement  in  their 
respective  areea  of  responsibility  and  to  provide  follow-up  to  assure  thet 
adequate  corrective  action  la  undertaken.  Figure  2  presents  a  typical  date 
flow  within  the  contractor's  R/M  ectivity.  Outputs  from  such  sn  activity 
include  tabulated  and  analyzed  failure  and  repeir  information,  and  most 
important,  recommendations  for  corrective  actions.  The  following  steps  should 
be  taken  by  the  contractor  on  all  date: 

(1)  The  R/M  engineers  screen  all  failure  end  repeir  reports  to 
determine  their  technical  validity,  and  then  classify  failure  information  ea 
tot 

(e)  Effect  of  the  failure  on  system  performance. 

(b)  Ceuse  of  failure. 

(c)  Responsibility  for  corrective  action. 

(2)  The  classification  activity  ia  supported  by  the  results  of 
analyzing  failed  perts  and  assemblies. 

(3)  Weak  links  are  identified,  and  MTBFa  and  VOTRa  are  calculated. 

(4)  Ra commendations  for  corrective  ectiona  to  eliminate  week  links 
are  generated  and  disaemineted  with  aubstantieting  information;  mathematical 
models  are  updeted. 

(5)  Most  importent,  re commend etiona  are  followed-up  to  determine  that 
auiteble  corrective  action  has  been  taken. 

Figure  3  presents  a  typical  corrective  action  flow  procase  thet  e 
contractor  should  undertake  to  effectuate  recommendations  for  corrective 
actions  submitted  by  his  R/M  activity.  A  plan  similar  to  this  should  be 
company  policy  and  ia  something  the  SPO  R/M  Monitor  will  look  for  in  epprsisal 
of  the  contractor's  policies,  procedures,  and  plans  for  implementing  an 
effective  R/M  ectivity.  This  process  is  setisfsctory  for  use  during  ell 
phases  of  the  contract,  and  is  presented  st  this  time,  within  the  comments 
pertaining  to  the  manufecturing  or  production  phsae,  es  e  matter  of 
convenience. 
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5.  Operational. or  Field  Evaluation  Phase  Requirements- 


e.  Data  Sources.  Obtaining  timely,  accurate,  end  complete  R/M  data  from 
the  field  is  probebly  the  most  difficult  to  achieve.  This  is  often  true  due 
to  incomplete,  or  lack  of,  early  planning  or  people  are  too  busy  attempting 
to  get  the  equipment  to  function,  which  ia  their  prime  mission.  Nevertheless, 
an  initiel,  we 11 -conceived  data  collection  plan  which  is  properly  coordinated 
with  all  concerned  should  reduce  data  collection  to  a  routine  activity. 

Data  sources  include  operational  logs,  contractor's  report  forma,  and 
report  forma  ea so elated  with  AFM  66-1;  namely,  AFTO  Forma  210  and  211,  Mainten¬ 
ance  Discrepancy /Product ion  Credit  Records;  snd  212,  Time  Compliance  Technical 
Order  Work  Record.  These  latter  forms  were  designed  primarily  to  serve  as 
source  documents  for  the  maintenance  data  collection  system  for  aircraft. 
Kowevar,  the  system  is  currently  being  modified  to  be  more  appropriate  for  usa 
by  all  Commands. 

b.  Nature  of  Data  and  Evaluation.  Tha  types  and  evaluation  of  field 
failure  and  repeir  date  are  much  tha  earns  as  dascribed  in  earlier  per a graphs. 
However,  greater  amphasis  ia  given  to  operational  malpractices  and  lncompeti^ 
bility  between  inplsnt  performance  specificationa  and  operational  specifics-, 
tions.  During  tha  operational  phase  of  a  given  program,  the  contractor's  R/M 
engineara  should  be  exerting  e  great  deel  of  effort  to  uncover  these  causes 
for  equip&ent  and  system  unavailability.  To  achieve  this  end,  the  R/M 
engineer  must  be  provided  with  both  quantitative  and  qualitative  information 
pertaining  to  a  failure.  Aa  an  example  of  this,  if  s  magnetron  ia  reported 
es  having  failed,  the  date  collected  should  answer  the  following  questions: 

What  parameter  vaa  out  of  specification?  What  whs  the  actual  reading?  What 
should  it  be?  According  to  what  document^  Provided  with  answers  to  these 
questions  the  R/M  engineer  should  be  able  to  methodically  evaluate  the  event 
by  checking  documents  auch  as  drawings,  performance  specificationa,  purchase 
specificationa,  procedures,  operating  manuals,  etc.,  end  comparing  this 
information  with  that  reported.  Given  adequate  information,  the  R/M  engineer 
esn  then  recommend,  if  necessary,  s  corrective  action  to  tha  appropriate 
activity  along  with  deflnltized  statements  sbout  any  existing  discrepancies. 

Field  failure  and  repair  reporta  elso  provide  inputs  for  tha  determina¬ 
tion  of  achieved  quantitative  reliability  and  maintainability.  From  theae 
documents  is  lifted  the  time  information  associated  with  times -to -failure  srd 
times -to -re pair  which  ere  usad  in  subsequent  calculations  of  MTBF,  KTTR,  and 
MDT.  Together  with  the  AFTO  Forms,  these  data  provide  inputs  to  the  SPO,  Using 
Commands,  snd  AFLC  for  the  determination  of  logistics  support,  types  and 
quantities  of  spares,  number  snd  gredes  of  maintenance  personnel.  To  give  a 
measure  of  the  worth  of  field  fsllure  end  repair  data  to  USAK,  the  following 
ia  quoted  from  page  1 -3  of  AFM  66- 1 ; 


7 


"1-1 4  Data  Collection  -  The  manhour  accounts  and  maintenance  data 
collection  procedurea  outlined  in  this  manual  are  continuously  assuming  greater 
importance  in  the  Air  Force.  (Note;  The  maintenance  data  collected  can  be 
adequately  processed  by  PC AM  methods.  This  method  affords  the  flexibility 
necessary  to  reedily  compile  data  essential  to  maintenance  management.)  Man¬ 
power  criterie  ia  to  be  oased  upon  this  data.  Besides  the  use  of  the  recorded 
elements  of  data  by  base  level  management,  the  same  data  is  used  extensively 
throughout  AFLC  management  levels,  for  many  purpoaea.  Some  specific  AFLC  uses 
are:  analysis  of  the  high  system  failures  end  the  high  system  consumer  of  man¬ 
hours  by  weapon  ayttkm;  identification  of  items  and  substantiation  for  product 
improvement  action;  analysis  of  established  inspection  requirements  and  a  basis 
te  adjust  knjp9ction  criteria;  analysis  snd  adjustment  of  tha  component  time 
change  cycles;  analysis  of  the  not  repairable  this  station  (NRTS)  listing, 
computation  of  spares  requirements  based  on  usege  in  lieu  of  the  SB  and  CR;  and 
aerospace  vehicle  selected  equipment  configuration  status  recording.  The 
significance  of  these  end  other  uses  of  the  data  for  management  throughout  the 
entire  materiel  function  maxes  it  imperative  that  elements  recorded  be  accurate, 
that  quality  data  is  accumulated-  In  vi®w  of  this  large-scale  AFLC-vide  usage 
of  the  data,  in  addition  to  base-level  management  usage,  it  is  obvious  that  data 
accuracy  and  coverage  15  ci  extreme  importance.  The  Chief  of  Maintenance  must 
continuously  act  to  insure  that  all  assigned  personnel  are  providing  100$ 
coverage  and  accurecy  in  these  data  recordings  He  must  also  insure  that  work 
center  supervisors  are  checking  input  data  on  the  original  documents,  each 
data,  for  accuracy  and  completeness.  In  addition,  commodity  information  will 
be  used  fcr  supply  consumption  reporting,  by  programs  and  master  repair  schedule. 
Accounting  and  budget  dats  is  alao  being  obtained  from  this  deta." 

It  is  extremely  important  that  the  contractor’s  reporting  system  is 
compatible  with  the  requirements  of  AFM  66-1  This  will  afford  ease  in  the 
transition  from  a  contractor -maintained  site  to  USAF  (blue-suit)  operation 
(usually  prior  to  Cat  II  tests)  Since  AFM  66-1  is  m  great  detail,  it  will 
be  well  for  each  SPO  R M  Monitor  to  review  it  to  become  familiar  with  its 
general  content  and  requirements. 
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SECTION  V 

RELIABILITY  DECISION  MAKING  -  CONSTRUCTION  AND  APPLICATION 
OF  PROBABILITY  OF  ACCEPTANCE  CURVES 


FOREWORD 


Current  statistical  literature  gives  little  guidance  in  choosing  between 
various  statistical  methods  and  cnoosing  ’’proper"  levels  of  risk.  These 
considerations  are  mostly  left  to  the  unaided  judgement  of  tne  decision 
maker . 

Tr.is  section  takes  several  statistical  procedures  and  places  them  within 
the  same  overall  framework,  in  order  to  provide  a  base  for  objectively  choos¬ 
ing  between  them.  The  reader  is  also  directed  to  Probability  of  Acceptance  curves 
as  an  aid  to  selecting  producer  and  consumer  risk  levels.  Finally,  a 
tabulation  of  the  advantages  and  disadvantages  of  each  of  the  various  methods 
is  presented. 
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Chapter  1 
INTRODUCTION 


This  section  has  a  three-fold  objective*  (l)  to  give  ESD's  position 
on  current  methods  for  selecting  statistical  accept/reject  criteria  in 
order  to  reach  decisions  on  equipment  reliability;  (2)  to  discuss  recur¬ 
ring  demonstration  problems  (unique  to  ESD  procurements)  and  certain 
statistical  methods,  not  available  in  current  statistical  literature,  for 
dealing  with  these  problems;  and  (3)  to  present  those  mathematical  con¬ 
cepts  and  methods  that  must  be  understood  by  those  responsible  for  the 
successful  management  and  implementation  of  reliability  programs. 

The  need  for  this  section  arises  from  the  fact  that  "statistical 
decision  theory"  is  a  comparatively  new  and  rapidly  changing  field.  Thus, 
there  is  no  single  document  that  one  can  select  to  extract  tbe  sam$  infor¬ 
mation  given  here.  Rather,  a  laborious  and  time  consuming  .study  of  many 
forbidding  treatises  is  required  and,  even  then,  one  is  likely  to  encounter 
difficulty  in  piecing  together  isolated  results  which  might  apply  to  a 
given  situation. 

Furthermore,  several  reliability  demonstration  specifications  are  in 
the  the  process  of  being  revoked  and  replaced  by  a  single  military  standard 
(which,  at  this  writing,  has  not  as  yet  received  a  numerical  designation). 

The  new  Standard  is  based  upon  ideas  explained  in  this  section  and  con¬ 
tains  numerous  test  plans  (decision  rules)  for  demonstrating  equipment 
reliability.  In  order  to  use  the  Standard,  one  must  consider  several 
things,  such  as:  Mean-Time-Between-Failure  (MTBF)  requirements  and  asso¬ 
ciated  failure  definitions;  the  intended  operational  environment  and 
associated  stress  testing  that  is  warranted;  the  amount  of  test  time  that 
may  be  Plotted  and  associated  risks  that -may  be  tolerated  in  making  accept/ 
reject  decisions. 

Since  our  subject  is  technical,  one  must  become  familiar  with  certain 
concepts  upon  which  the  entire  subject  is  based.  If  the  reader  has  diffi¬ 
culty  absorbing  these  concepts,  he  should  keep  in  mind  that  most  often  the 
trouble  is  caused  by  the  language  that  is  used.  Habitually,  ordinary  words 
are  used  in  special  senses;  a  practice  that  is  convenient  if  one  has  mas¬ 
tered  their  use,  but  disconcerting  if  one  has  not.  In  any  case,  the  language 
must  be  endured,  since  once  it  has  developed  only  minor  changes  are  feasible. 

This  section  is  written  with  these  thoughts  in  mind.  With  the  exception 
of  Chapter  3,  "Technical  Considerations",  #eat  pains  are  taken  to  present 
concepts  and  methods  with  a  minimum  of  tecnnical  jargon  and  notation.  In 
Chapter  2,  particularly,  the  style  is  deliberately  wordy;  and  although 
a  certain  amount  of  special  vocabulary  and  symbolism  Is  required  for  ef¬ 
ficiency  of  thougkt,  only  those  which  are  considered  essential  to  explain 
key  results  are  used,  however  convenient  it  might  be  to  do  otherwise. 


Similarly,  certain  significant  topics  are  deliberately  excluded  (such 
as  "methods  for  confidence  interval  estimation'*)  because  of  the  many 
technical  details  required  to  give  them  adequate  explanation,  and  because 
knowledge  of  these  topics  is  not  considered  essential  for  the  results 
obtained  herein.  While  the  discussion  of  related  topics  may  have  broadened 
the  scope  of  this  section  the  risk  of  creating  confusion  on  the  part  of 
the  uniniated  reader  demanded  that  such  topics  be  omitted. 
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Chapter  2 

MANAGEMENT  CONSIDERATIONS 


1.  Background  -  Measuring  Reliability.  In  studying  the  behavior  of  a 
physical  process*  a  conceptual  (mathematical)  model  is  sought  which  will 
bring  together  certain  observed  variables  in  order  to  derive  meaningful 
and  unambiguous  etatemente  which  gain  universal  acceptance.  Hopefully, 
tne  establishment  of  an  idealized  mathematical  structure  permits  a  quan¬ 
titative  characterization  of  the  proceee  that  enables  us  to  predict  its 
future  behavior  ae  a  reeult  of  analyzing  data. 

The  diecipline  of  Reliability  Engineering  has  established  such  a 
structure,  that  is,  a  quantitative  baeis  for  measurement,  and  is  con¬ 
tinuing  to  build  upon  it.  Indeed,  some  define  Reliability  as  the  science 
that  predicts  mathematically  the  failure  behavior  of  a  particular  device. 
Quite!  commonly,  one  finde  Reliability  Engineers  involved  in  "distribu¬ 
tion  theory"  or  the  application  of  probabilistic  laws,  in  order  that  they 
may  describe  the  interactions  of  parts,  equipments,  and  systems. 

It  didn’t  start  out  thie  way.  In  the  beginning,  Reliability  was 
created  more  qualitatively.  As  recently  ae  the  early  1950’s,  there  were 
severe  arguments  over  the  definition  of  Reliability  -  some  held  that  it 
was  a  "feeling"  that  a  collection  of  equipmente,  people,  etc.,  would 
yield  deeired  resulte.  At  thie  time  there  was  concentration  on  data 
collection,  classification,  and  engineering  analysis  of  data  to  lessen 
tr.e  frequency  of  failures.  One  may  tersely  describe  the  underlying  at¬ 
titude  that  prevailed  in  thie  period  as  a  "build-fly -fix"  philosophy. 

Reliability  entered  a  "management  era"  in  the  mid-1950's,  during 
wnich  time  there  was  a  struggle  to  develop  etandard  terminology,  and 
techniques  for  organizing  a  reliability  program.  Major  accomplishments 
during  this  period  were  studies  performed  by  Aeronautical  Radio,  Incorp¬ 
orated  (ARINC),  which  eventually  led  to  the  development  of  the  first 
Reliability  prediction  technique  for  airborne  bombing  navigation  equip¬ 
ment,  and  Radio  Corporation  of  America  (RCA),  which  presented  the  inter¬ 
action  between  operating  s trees  levele  and  part  failure  rates.  These 
efforts  provided  a  basis  for  designing  Reliability  into  electronic  equip¬ 
ments  and,  when  coupled  with  the  fact  that  exotic  demands  on  the  per¬ 
formance  of  electronic  equipments  were  having  such  disastrous  effects  on 
their  reliability,  made  it  imperative  that  acquisition  philosophy  change 
to:  "build-1 t-right-the-first-time. " 

Moreover,  complexity  of  electronic  equipment  was  continually  in¬ 
creasing,  making  it  mandatory  that  methods  for  quantitative  analysis  be 
developed.  Efforts  toward  this  end  culminated  in  the  AGREE*  Report, 

1957,  issued  by  the  Assistant  Secretary  of  Defense,  a  milestone  in  Re¬ 
liability  documentation.  Several  military  specifications  arose  at  this 
time  requiring  that  quantitative  Reliability  indices  be  establiehed  and 
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demonstrated  before  committment  to  operational  usage. 

Thus,  by  1960,  Reliability  emerged  as  a  new  discipline.  Separate 
military  staffs  and  industrial  departments  appeared  on  the  scene,  as 
well  as  text  books  and  research  reports  all  apparently  aimed at  re  m- 
ing  the  mathematical  aspects  of  the  theory  of  Reliability.  Emphasis 
was  placed  upon  statistical  methods  for  estimating  the  degree  of  reli¬ 
ability  that  was  being  achieved. 

Yet.  as  the  use  of  mathematical  concepts  increased,  the  amount  of 
special  terminology  and  notational  devices  needed  to  intelligently  dis¬ 
cuss  the  subject  also  increased,  and  those  not  fully  engrossed  in  the 
field  became  less  familiar  with  its  progress.  It  became  apparent  that 
Reliability  was  being  measured  without  regard  to  several  important  factors, 
such  as  the  intended  use  of  the  equipment,  cost  of  testing,  schedul  , 
and  equipment  characteristics.  In  many  cases,  the  statistical  refinm 
were  excessive  causing  unnecessary  delays  in  decision-making  or,  w..at  is 
worse  waiving  of  all  demonstration  requirements.  On  the  one  hand,  leng  y 
tests  had  been  specified,  by  other  than  program onagers,: in  order  to 
"prove"  reliability,  and  on  the  other  hand,  management  had  created  schedules 
which  Wvv>'e  incompatible  with  such  testing. 

There  is  perhaps  only  one  way  to  avoid  this  situation,  that  is,  if 
program  managers  ask  themselves  one  important  question:  Why  are  we  meas¬ 

uring  Reliability?"  Related  questions  are  "Do  we  need  close  estima.es  of 
actual  numerical  values,  or  is  it  sufficient  to  simply  give 
assurance  that  these  values  are  above  minimum  levels.  What  are  these 
minimum  levels  and  how  much  test  time  may  reasonably  be  devoted  to  this 
eff orf?  How  does  adding  or  subtracting  test  time  affect  our  assurance 
that  such  requirements  have  been  met?" 

2.  Measures  of  Reliability.  Usually,  the  above  questions  are  not  dif¬ 
ficult  to  answer  if  quantitative  measures  for  the  reliability  of  the 
equipment  or  system  under  consideration  have  been  established.  Thus,  be¬ 
fore  explaining  methodology  for  making  this  analysis,  it  is  necessary  to 
explain  certain  measures  which  have  been  found  to  apply  to  large  classes 
of  electronic  equipment.* 


*  Readers  with  previous  experience  in  this  subject  may,  of  course,  skip 
oaracraoh  2..  Since  this  statement  may  arouse  their  curiosity,  it  may  be 
appropriate  to  set  forth  what  we  have  tried  (and  not  tried)  to  accomplish. 
We  have  attempted  to  give  those  with  no  previous  experience,  an  intuitive 
prasp  of  one  method  for  establishing  quantitative  reliability  measures  in 
order  to  explain  certain  problems  that  arise  in  designing  demonstration 
tests.  Hence,  this  is  not  a  precise,  systematic,  or  all-ioclusive  devel¬ 
opment  -  experts  may  detect  areas  of  over-simplication,  or  even  errors-, 
although,  of  the  latter,  none  were  intentionally  included. 
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In  what  follows,  it  is  helpful  to  distinguish  between  two  types  of 
electronic;  devices.  One  would  be  &  device  that  ii  designed  for  contin¬ 
uous  operation  and  comprised  of  two  or  more  replaceable  sub-units  (or 
parts).  It  is  also  designed  so  that  a  failure  of  any  of  these  sub-units 
(parts)  causes  the  device  to  be  inoperable,  and  restoration  is  made 
simply  by  replacing  the  malfunctioning  sub-unit.  For  this  type  of  device, 
it  is  not  surprising  that  most  people  are  concerned  about  the  question 
"how  long,  on  the  average,  does  it  operate  between  consecutive  failures?" 
This  question  is  usually  answered  by  giving  its  "mean-time-be tween-failure 
(MTBF").  On  the  other  hand,  some  people,  not  quite  so  far-sighted,  may 
be  apprehensive  about  the  very  next  failure  and  may  phrase  their  question 
"how  long,  on  the  average,  before  the  first  failure?"  This  is  really 
the  uame  question  as  before  (but  with  a  different  orientation)  and  re¬ 
ceives  the  same  answer  except  that  it  is  verbalized  as  "me an- time- to- 
failure  (MTTF)",  This  response  is  a  commonplace  when  dealing  with  a 
second  type  of  device,  namely,  one  with  no  replaceable  sub-units  besides 
itself.  Thus,  failure  of  ths  device  means  its  "death",  and  it  is  for 
this  reason  that  MTTF  is  sometimes  referred  to  as  "Mean  Life". 

It  is  only  natural  to  speak  in  terms  of  devices  of  the  first  kind 
since  they  occur  most  frequently  in  ESD  procurements.  Two  important 
advantages  accrue  from  this  choice.  One  is  that  the  single  word  "equip¬ 
ment"  may  be  employed  throughout,  when  referring  to  these  devices.  The 
otner,  perhaps  more  important,  is  that  the  reader  is  never  thinking  of 
the  wrong  example.  Thus,  the  expressions  MTTF  and  Mean  Life  will  not 
be  encountered  again. 

That  MTBF  is  measurable  is  the  theme  of  ensuing  paragraphs.  Its 
development,  however,  depends  on  a  number  of  assumptions.  The  first  and 
most  far-reaching  assumption  that  shall  be  made  is  that  MTBF  is  a  fixed 
quantity,  say  5,  200,  or  10,000  hours,  defending  on  the  design  of  the 
equipment  (and,  possibly,  its  environment).  Naturally,  ons  could  not 
expect  MTBF  to  be  a  fixed  quantity  for  the  entire  life  of  the  equipment. 
The  following  curvet 


Failure 

Rate 


koswn  as  the  bath-tub  curve,  illustrates  that  equipments  exhibit  their 
best  "failure  behavior"  after  certain  manufacturing  and  design  errors  are 
removed  (during  what  is  commonly  called  the  "burn-in  stage")  and  before 
rather  extensive  replacements  become  required  as  a  result  of  age  (or  wear 
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out).  The  period  between  the  burn-in  etage  and  the  wear-out  stage  is 
called  the  "useful  life",  and  it  ie  here  that  MTBF,  as  a  fixed  value, 
takes  on  its  true  meaning.  The  expreesion  "failure  rate"  is  defined  as 
the  reciprocal  of  MTBF  (i.e.,  if  the  MTBF  is  200  hours,  we  expect  fail¬ 
ures  to  occur,  on  the  average,  once  each  200  houre  eo  that  i/200  is  termed 
the  failure  rate). 

Another  important  assumption  is  that  failures  occur  at  random.  For 
those  not  familiar  with  probability  theory,  ,thie  may  eeem  to  contradict 
the  assumption  that  MTBF  ie  a  fixed  value  (during  the  useful  life  of  the 
equipment)  but  there  is  no  contradiction.  Consider,  for  example,  the 
occurrence  of  a  "eeven"  upon  the  toseing  of  dice.  Although  thie  occurrence 
ie  quite  unpredictable  on  a  single  toee  (therefore  random),  most  would 
agree  that  for  the  case  of  unloaded  dice  betting  against  such  an  occurrence 
on  repeated  tosses  would  cauee  ue  to  win,  on  the  average,  5  out  of  6 
timee.  Thus  the  "mean-toeees-between-sevensr*  may  be  coneidered  a  constant, 
in  spite  of  the  fact  that,  occasionally^ we  experience  3  or  4  "sevens" 
consecutively  and  the  abeence  of  a*seven*on  LO  or  11  repeated  toeses  is 
not  uncommon. 

Hwnce,  long  runs  of  failure-free  operation  average  with  short  runs  of 
repeated  failures  to  give  an  overall  value  that  is  considered  the  equip¬ 
ment  failure  rate.  One  may  be  tempted  to  take  the  number  of  failures 
that  occur  and  divide  it  by  the  total  operating  time  to  obtain  an  estimate 
of  the  failure  rate  just  ae  one  usually  divides  the  number  of  toesee  which 
yield  a  "seven"  by  the  total  number  of  tosses  to  help  decide  whether  dice 
are  biased  with  respect  to  the  attribute  "eeven".  But  one  does  not  have 
to  be  an  expert  in  statietice  to  realize  that  thie  procedure  gives 
erroneous  results  quite  often  due  to  "chance  fluctuations",  particularly 
if  small  amounts  of  data  are  collected.  In  fact,  one  can  say  that  the 
main  role  of  statistics  is  to  develop  procedures  for  coping  with  thie 
problem.  Most  likely,  it  haa  already  occurred  to  the  reader  that  the 
procedure  just  explained  works  quite  weii  if  "enough"  data  is  collected, 
that  is,  the  weii  known  "law  of  averages"  begins  to  operate  after  a  while 
leaving  little  doubt  in  the  minds  of  reasonable  people.  Thus,  if  5 ,000 
"sevens"  occur  in  10,000  tosses,  one  does  not  need  advice  from  experts 
to  decide  that  the  dice  are  biased. 

So  far,  the  implication  has  been  made  that  MTBF  is  a  measure  of 
reliability,  but  the  relation  between  the  two  or,  more  precisely,  the 
effect  that  MTBF  has  on  reliability  has  not  been  eta ted.  To  do  this 
requiree  a  definition  of  the  word,  a  matter  that  has  been  neglected  until 
now.  Therefore,  without  further  ado,  reliability  is  defined  as  "the 
probability  of  faiiure-free  operation  for  a  given  amount  of  operating 
time".  This  ie  a  good  working  definition  which  gets  the  point  across 
that  reliability  ie  a  probabilistic  notion  as  weii  as  a  function  of  time, 
and  it  ehouid  not  shock  anyone  that  the  resulting  probability  is  quite 
dependent  upon  the  MTBF  of  the  equipment  under  coneideration. 
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Thie  is  all  that  shall  be  said  about  "measures  of  reliability"  since 
Section  VI,  "Verification  of  Quantitative  Reliability  Requirement e" , 

give  a  more  detailed  discussion  of  thie  matter.  In  that  document,  the 
problem  of  measuring  reliability  ueing  epecific  methods  ie  discussed*  here, 
the  problem  is  to  develop  methodology  for  choosing  between  various  methods. 

3.  The  Deponatratlon  Problem.  The  preceding  discuesion  elready  gave 
hints  as  to  the  difficulties  aeeociated  with  drawing  inferences  about 
reliability  ae  a  result  of  a  demonetration,  namely,  that  almost  any 
feilure  behavior  could  be  consietent  with  any  value  of  MTBF.  Thie  problem 
becomes  acute  when  short  periode  of  teeting  are  involved,  but  is  chron¬ 
ically  present  even  when  long  periods  of  testing  are  permitted, 

flLong"  and  "short"  are,  of  couree,  relative  terms,  which  mostly 
depend  upon  MTBF  requirements  for  their  meaning.  For  example,  500  houre 
is  considered  a  lengthy  test  for  a  10  hour  system,  and  quite  brief  for 
testing  equipments  with  MTBF1 a  of  1,000  hours  or  more.  For  production 
procurements  5,000  hours  may  not  be  considered  a  long  demonstration  if 
enough  models  can  be  tested  simultaneously  and  their  operating  timee  and 
quantities  of  failures  are  combined.  (This  is  permitted,  statistically 
speaking,  as  long  as  enough  time  ie  accumulated  on  each  equipment  to  in¬ 
sure  that  they  have  progressed  beyond  the  burn-in  etage.) 

Unfortunately,  ESD  is  mostly  confronted  with  non-production  procure¬ 
ments  end  high-order  MTBF  requirements  (say  500  houre  or  more).  In  fact, 
one  must  usually  keep  in  mind  that  only  about  720  calendar  hours  are 
evailable  each  month.  It  becomes  extremefy  difficult  to  design  a  test 
‘hat  is  long  enough  so  that  eome  failures  can  be  expected  to  occur. 

Clearly,  little  Is  gained  by  observing  failure-free  operetion  in  a  given 
operating  time  if  one  would  not  even  expect  unsatisfactory  equipment  to 
neve  any  failures  in  that  time.- 

Still  it  appeare  that  one  is  forced  to  make  decisions  about  MTBF  based 
upon  the  number  of  failures  that  occur  (if  any)  in  a  given  amount  of  oper¬ 
ating  time.  It  is  easy  to  eee  that  the  Air  Force  cannot  establish  a  decision 
criterie  that  always  rejects  equipment  with  MTBF'e  that  are  inconsistent 
with  specified  requirements,  since  euch  a  criteria  would  aleo  reject 
jatlsfactqry  equipments  too  often.  More  specifically,  there  are" two  types 
of  risks  present.  They  warrant  epecial  name3  because  they  must  be  care¬ 
fully  scrutinized  in  any  statistical  decision  rule  of  this  kind.  These 
are : 


Air  Force  Risk:  The  probability  that  unsatisfactory 
equipment  will  be. accepted. 

Producer  Risk:  The  probability  that  satisfactory 
equipment  will  be  rejected. 
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X. 


Chapter  3  of  this  pamphlet  gives  detailed  methods  for  quantifying 
theee  risks  and  keeping  them  below  prescribed  levels  when  making  eccept/ 
reject  demons,  and  Chapter  4  gives  ar  analytic  discussion  of  how  low 
risKS  affect  the  required  test  time.  It  becomes  apparent  there  that 
riske  cannot  be  esteblished  without  considering  the  amount  of  test  time 
that  is  available.  Hence  there  cannot  be  a  single  decision  rule  that 
appliee  to  all  procurements. 

Another  closely  related  problem,  but  one  which  has  little  to  do  with 
probabilistic  concepts,  concerns,  the  MTBF  requirement.  It  has  become 
customary  to  cite  e  single  figure,  say  500  hours,  as  the  minimum  acceptable 
value,  and  then  require  that  this  value  be  demonstrated.  Such  a  require¬ 
ment  may  be  unreasonable  from  two  distinct  viewpoints.  From  a  P^tica1 
view  it  does  not  seem  plausible  that  500  hours  is  acceptable  but  499  is 
not  ’  Moreover,  those  who  are  statistically  oriented  shudder  at  the 
thought  of  eatisfactory  and  unsatisfactory  values  being  separated  a 
hair-line,  a  situation  that  is  certain  to  play  havoc  with  the  risks  . 

It  turne  out  (ae  pointed  out  in  Chapter  3)  that  under  these  conditions, 
if  the  Air  Force  maintains  its  risk  at  10%,  the  producer  s  risk  must  be 
90%  tuAt  ie  both  risks  must  add  to  100%.  There  is  perhaps  only  one  way 
to  avo.i  th.se  high  risks.  If  operational  requirements  call  for  a  500 
hour  MTBF,  then  this  sust  be  the  value  that  is  labeled  satisfactory  ,  and 
should  be  used  for  determining  the  producer’s  risk  (how  this  determination 
is  mede  shall  be  covered  in  Chapter  3.)  However,  those  values  just  below 
500  hours  cannot  be  termed  unsatisfactory ,  since  it  appears  more  reason¬ 
able  to  specify  an  "MTBF  lower  bound",  say  450  hours,  such  that  it  becomes 
a  matter  of  concern  if  the  MTBF  should  be  this  low  (due  to  the  deteri¬ 
orating  effect  on  operational  effectiveness).  Thus,  MTBF  s  less  than  or  • 
equal  to  450  hours  are  termed  "unset! sfac to ry"  and  the  Air  Force  risk  may 
be  computed  based  on  this  number  rather  than  500.  This  number  (500;  cou 
be  called  the  "MTBF  upper  bound"  or  "MTBF  objective  .  Here  too,  it  is 
ludicrous  to  eay  that  451  hours  is  satisfactory,  that  is,  all  that  can  be 

said  ebout  values  greater  than  450  but_less  than  5QQ  is  n^thia  kind 
neither  satisfactory  nor  unsatisfactory.  Under  a  stipulation  of  this  kind 
it  is  possible  to  hold  both  risks  at  relatively  low  levels.  The  details 
of  doing  so  shall  be  explained  in  Chapter  3. 

One  last  problem  worth  mentioning  because  it  is  frequently  not  ad¬ 
equately  considered,  involves  the  definition  of  "failure".  This  problem 
is  not  eolved  by  reliability  specialists  alone— there  are  both  engineering 
and  operational  requirements  to  be  considered.  Similarly,  how  the  equip¬ 
ment  is  to  be  exercised  during  the  demonstration  is  a  factor  that  requires 
some  thought  if  the  demonetretion  is  to  heve  any  relation  to  the  intended 
uee  of  the  equipment.  At  this  point,  it  should  be  noted  thet  the  intended 
environment" waslsft  out  of  our  definition  of  Reliability,  in  order  not  to 
clutter  up  our  math  model.  This,  of  course,  must  be  compensated  for  by 
thoee  designing  or  specifying  the  conditions  of  test. 
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4,  Choosing  Acoeot/fre !eot  Criteria  -  Uee  of  Probability  of  Accuptunce 

Ciprrea.  Onoe  meaningful  requirement*  have  been  etipulated,  it  re- 

maine  to  e»tabll*h  an  accept/reject  criterion  for  the  demonstration. 

There  exist  military  standards  and  epecificatione  which  give  specific 
accept/reject  criteria  to  be  used  for  demonstrating  reliability  of  elec¬ 
tronic  equipment.  The»e  may  be  hslpful  In  some  caeee,  but  cannot  be 
ueed  indi* criminate ly.  They  are  baeed  upon  procedure©  which  attempt  to 
hold  both  Air  Force  and  producer  rieke  below  certain  levele,  but  all 
involve  a  certain  amount  of  operating  time  before  decisions  may  be  made. 
Neturally,  one  should  choose  the  one  that  gives  the  leaet  risk  consistent 
with  the  amount  of  time  that  may  be  expended  for  demonstration  purposes. 

Another  method  exists  for  chooeing  between  available  acfcept/reject 
criteria.  In  Chapter  4  the  construction  and  application  of  "probability 
of  acceptance  curve*"  (PA  curves)  ie  described.  For  any  particular  cri¬ 
teria  that  we  select,  these  curve©  will  give,  at  a  glance,  the  probability 
that  squipment  with  various  levele  of  trus  MTBF  will  be  accepted— that  is, 
paee  the  test.  Hence,  the  producer  can  derive  design  goal©  which  give 
near-certainty  of  being  aocepted  or,  at  least,  very  low  rieka.  On  the 
other  hand,  the  probability  of  aocepting  low  quality  equipment  is  elso 
made  apparent,  thereby  causing  responeible  Air  Force  official©  to  take  an 
active  interest  also. 

For  example,  suppoee  that  the  MTBF  lower  bound  ie  400  houre  and  the 
MTBF  upper  bound  is  500  houre.  Then,  as  previouely  noted,  values  of  true 
MTBF  leee  than  or  equal  to  400  houre  are  considered  unsatisfactory  and 
values  of  true  KTBF  greater  than  or  equel  to  500  hours  are  considered 
satisfactory.  Nov,  1st  ue  suppose  that  two  modele  may  be  placed  eimul- 
taneouely  on  teat,(yith  the  agreement  that  their  operating  timee  and 
numbers  of  failure®  shall  be  combined)  but,  either  the  cost  of  testing  or 
"tight"  echedulss  prohibits  more  than  500  hours  of  reliability  testing. 
Hence,  the  number  of  feiluree  permitted  for  acceptance  purposes  must  be 
decided  upon.  For  simplicity,  let  ue  try  to  decide  between  0,  1,  or.  2, 
by  using  PA  curves. 

Figure  i  (see  next  page)  shows  the  PA  curvee  for  0,  1,  2  fei lures 
allowed,  respectively,  in  500  houre  of  test.  Looking  at  Curve  A  we  see 
that  even  if  the  producer  has  designed  equipment- with  true  MTBF  equel  to 
1,000  houre  (400  houre  above  the  requirement),  the  probability  of  accept¬ 
ance  ie  only  about  60%;  hencs,  thie  test  (0  failures  allowed)  may  be 
considered  too  eevere.  On  the  other  hand,  Curve  B  shove  that  equipments 
with  true  MTBF  equal  to  250  (250  hours  lese  than  required)  bare  e 
40%  chance  of  being  aocepted;  hence,  thie  test  (l  failure  allowed)  may 
be  considered  too  lenient.  Clearly,  the  producer  would  be  eatiefied  with 
2  failures  allowed  (Curve  C)  since  this  rule  would  give  better  than  90% 
probability  of  aoceptance  for  any  value  equal  to  or  better  than  the  500 
hour  requirement.  But  such  a  rula  aliowe  400  hour  equipments  (which  ere 
considered  unsatisfactory)  better  than  80%  chance  * of  being  aocepted. 


Currt  A: 
Curve  B: 
Cunrf  C; 


500  test  hour*,  0  fad  lure 0  allowed 
"  "  "  ,  1  failure 

"  "  "  ,  2  failure* 


Tioaia  1:  PA  CURVES  FOR  500  HCURS 
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Obviously,  non*  of  the**  decieion  rule*  will  satisfy  both  parties. 
Either,  more  tot  time  must  be  obtained,  or  upper  and  lower  bound  values 
must  be  re-examined.  For  example,  the  decieion  ruia  depicted  in  Curve  B 
might  be  reasonable  if  ths  lower  bound  value  were  200  rather  than  400 
houre.  In  other  words,  if  the  procuring  agency  determined  that  values 
of  MIBF  between  200  and  500  hours  would  not  drastically  affect  opera¬ 
tional  effective neaa,  the  allowance  of  i  failure  in  500  houre  of  teet 
gives  reasonable  aesurance  to  both  parties. 

However,  suppose  teet  tims  could  be  increased  to  1,000  hours  (pos¬ 
sibly  by  doubling  the  number  of  models  eimultaneously  tested,  or  as  a 
result  of  rescheduling).  Let'e  now  look  at  PA  curves  for  0,  1,  2,  and 
3  failures  allowed  (see  Figure  2,  next  page)  in  1,000  houre  of  test. 

Using  ths  same  reasoning  ae  before,  it  appears  that  Curve  F  (2  fail¬ 
ures  in  1,000  hours)  comes  closest  to  satisfying  both  parties,  although 
both  would  have  some  reservations  about  such  a  decision  rule. 

Aii  examples  used  thue  far  have  assumed  that  a  fixed  amount  of  test¬ 
ing  would  be  conducted  and  a  certain  amount  of  failures  allowed  for 
acceptance,  say  x,  with  the  implication  being  that  if  x  +  1  or  more  fail¬ 
ures  occur  in  that  time  then  a  reject  decision  would  be  made.  It  was 
also  assumed  that  test  time  was  extremely  limited,  relative  to  the  MTBF 
requirement;  that  is,  in  the  first  case  (5OO  hours)  we  could  only  test 
one  multiple  of  the  MTBF  requirement  (also  called  "MTBF  upper  bound"  or 
MTBF  objective,")  and  in  the  second  case  (1,000  hours)  only  2  multiples 
of  MTBF  testing  was  permitted. 

When  3  or  more  multiplee  of  MTBF  testing  are  permitted,  risks  may  be 
held  much  lower  and,  in  fact,  earlier  decision  points  may  be  stipulated. 

To  see  this  mors  clearly,  obeerve  that  Curve  A  of  Figure  1  shows  that 
after  500  houre  of  testing,  if  no  failures  occurred,  the  probability  of 
acceptance  for  equipments  with  MTBF  less  than  or  equal  to  400  hours  is 
at  most  28%.  This  "worst  case"  probability  may  be  called  the  Air  Force 
risk,  representing  the  "probability  that  unsatisfactory  equipment  will 
be  accepted."  What  we  are  saying,  then,  is  that  if  the  Air  Force  accepts 
on  the  basis  of  0  failures  in  500  hours,  there  is  a  28%  risk. 

Now,  if  risks  below  (say)  3<#  can  be  tolerated,  the  Air  Force  could 
accept  at  this  point.  But  what  about  rejection?  Another  glance  at 
Figure  1  (Curve  B,  thle  time)  shows  that  equipments  with  MTBF’e  of  500 
hours  or  more  have  at  leaet  72%  chance  of  passing,  if  1  failure  were 
allowed.  This  may  be  reinterpreted  as  follows!  Such  equipments  have  only 
100%  -  72%  -  28%  chance  of  having  2  or  more  failures  in  500  hours.  This 
again  is  a  "worst  caes"  probability,  and  may  be  called  the  producer’s 
risk. 

Hence,  after  500  houre  of  test,  the  Air  Force  could  accept  if  0 
failures  occurred  and  rajec^  if  2  failures  occurred,  with  (at  most)  28% 
risk  to  both  parties.  Of  course,  if  exactly  1  failure  occurred  in  that 
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PA  Curves  for  1,000  Hours. 


Figure  2. 


time,  no  decision  ecu  id  be  made,  but  the  test  could  be  continued  and  at 
various  points  during  the  test  (say,  at  each  multiple  of  MTBF )  the  same 
kind  of  determination  made.  Eventually,  a  point  would  be  reached  where 
the  number  of  failures  specified  for  acceptance  is  one  less  than  the 
number  of  failures  specified  for  rejection,  with  both  risks  bsing  the 
same. 


The  ideas  explained  in  the  preceding  paragraphs  come  under  the  name 
"sequential  testing",  and  specific  procedures  for  devising  such  tests 
are  detailed  in  ESDP  80-5.  We  may  conclude  our  comments  here  by  saying 
that  sequential  tests  should  be  used  whenever  the  amount  of  test  hours 
permitted  is  sufficient  since  prescribed  levels  of  risk  ars  never  vio¬ 
lated  and  yet,  more  often  than  no early  decisions  are  reached.  Psrhaps 
this  is  caussd  by  a  tendency  for  equipments  to  be  "extremely  good"  when 
they  meet  requirements  and  "extremely  bad"  when  they  don*  t.  Of  course, 
if  sufficient  test  time  is  not  available,  one  must  resort  to  the  fixed¬ 
time  approach. 

The  chapters  that  follow  shall  first  cover  the  basis  for  various 
decision  rules,  some  of  which  have  been  briefly  explained  in  this  chapter, 
and  then  give  a  comparative  analysis  of  these  rules,  citing  advantages 
and  disadvantages  of  each  for  different  applications. 
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Chapter  3 

TECHNICAL  CONST  EERATIONS 


I.  Pealgnlng.Teet  Criteria; 


a.  Introduction.  In  any  scientific  endeavor  gathering  of  data  is 
usually  a  meaningless  exerciee  unless  one  first  establishes  pertinent 
hypotheses  and  then  focuses  on  the  question  "how  well  do  these  facts 
fit  the  given  hypotheses?'1  In  other  words  "the  facts  usually  don1 1 
speak  for  themeelvee.  "  If,  in  turn,  the  data  is  quantifiable  and  the 
hypotheses  take  mathematical  form,  it  is  usually  a  simple  matter  to 
state  an  unambiguous  criterion  for  accepting  or  rejecting  the  given 
hypotheses.  However,  there  are  two  major  problems  associated  with  the 
establishment  of  an  accept/reject  criterion.  One  is  that,  h&ving  quanti¬ 
fied  the  data,  such  quantities  as  are  derived  from  tests  or  experiments 
appear  random  in  nature  (sometimes  called  “chance  fluctuations"),  that  ie, 
having  knowledge  of  previous  results  does  not  enable  one  to  make  precise 
and  meaningful  predictions  about  the  very  next  result.  The  other  is  that 
additional  methods  are  required  to  determine  how  good  or  how  bad  the 
nccept/reject  criterion  is.  Does  it,  for  example,  consistently  make 
correct  decisions?  How  do  we  define  "correct",  and  how  do  we  compare 
differing  criteria  that  may  be  contemplated?  It  would  be  nice  if  we  could 
order  them  in  euch  a  way  that  the  one  at  the  top  of  the  list  would  be  best 
to  use,  but  unfortunately  this  is  not  always  possible.  Still,  it  ie 
possible  to  cite  advantages  and  disadvantages  of  each  for  a  particular 
application,  and  then  give  the  rationale  for  the  choice  one  makes.  Here, 
several  choices  shall  be  presented  and  compared  in  this  manner.  Before 
doing  so,  it  is  necessary  to  give  a  general  discussion  of  the  process 
which  quantifies  data  and  makes  decisions  on  the  basis  of  the  collected 
data.  In  what  follows  this  process  is  called  a  "deci ei on-rule" .  No 
attempt  shall  be  made  to  give  a  systematic  theory  of  decision  rulee, 
since  this  requires  more  tims  and  space  than  is  at  our  disposal.  Rather, 
it  is  intended  to  reflect  the  "modus  operandi"  of  decision  making. 


b.  Elements  of  Decision  Rules.  Most  decision  rules  begin  by  making 
certain  assertions  (not  to  be  confused  with  hypotheses)  concerning  the 
nature  of  the  situation  that  is  involved.  Theee  assertions  (called  axioms) 
are  the  result  of  observation  (experience)  and  usually  are  so  basic  that 
attempts  to  justify  them  meet  with  frustration  and  ultimately  with  despair. 
(For  example,  the  basic  laws  contained  in  Attachment  1.)  Happily,  few  ask 
to  have  them  justified,  that  is,  they  are  accepted  by  most  people  as  being 
self-evident.  These  axioms  always  contain  undefined  concepts  which  take 
on  meaning  when  a  particular  application  is  called  for,  Ths  next  step  is 
to  derive  certain  theorems  from  theee  axioms  by  means  of  logical  argument. 
Finally,  these  theorems  are  compared  with  obeervsd  data  and  reinterpreted 
in  order  to  establish  a  rule  of  procedure  which  telle  us  whether  to 
accept  or  reject  certain  hypothesee  or  to  continue  the  experiment  until 
sufficient  data  is  collected. 
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More  briefly,  a  dacislon  nils  contains  two  major  ingrediantsi 
fl)  a  mathematical  model  of  the  obaerved  phenomenon;  and  (2)  an  accept  - 
reject  criterion  which  takss  collected  data  and  reinterprets  statements 
concerning  the  model  into  statements  about  the  observed  phenomenon  .  These 
ideas  may  be  summarized  pictorially  aa  follows; 


DECISION  RUIZ  MACHINE 


PHASE  It  MATH  MODEL 


1  The  following  remark  ia  for  the  benefit  of  thoae  who  may  recognize  a 
similarity  between  the  following  discussion  and  "Tests  of  Hypotheses  ,  as 
covered  in  statistical  texts.  There,  (2)  is  thought  of  as  the  decision 
rule  itself,  and  (l)  remains  hidden.  One  reason  for  this  is  that  a  certain 
type  of  mathematical  modal  exists  (called  a  probabilistic  model)  which  is 
"a  model  of  a  mathematical  model."  This  permits  the  development  of  a 
specific  mathematical  modal  simply  by  maxing  inputs  to  the  more  general 
model.  These  inputs  may  be  stated  generally  (e.g. ,  a  density  function)  and 
the  ruls  of  procedure  can  simply  call  these  out  so  that  the  mathematical 
model  is  created  as  the  rule  of  procedure  ia  applied.  Whether  or  not  it  is 
reasonable  to  uss  the  same  rule  of  procedure  for  differing  mathematical  models 
Is  a  problem  which  has  not  aroused  much  interest  In  those  who  may  be  qualified 
to  solve  it.  Most  liksly,  ths  success  of  the  probabilistic  model  for  dealing 
with  the  problem  of  randomness  has  left  littls  choice  in  the  matter. 


c.  Thft  Probabilistic  Model.  The  decision  rules  that  will  be  pre¬ 
sented  here  employ  a  "probabilistic  model".  Such  a  model  contains  ba^lc 
axioms  which  are  given  in  Attachment  1  of  this  document.  It  should  be 
noted  that  the  main  feature  of  this  general  modal  is  the  aassrtion  that 
l  function  P  sxists  which  gives  the  probability  of  an  event  E  occurring 
for  any  event  E  that  may  occur.  But  what  properties  must  the  observed 
physical  situation  satisfy  in  order  to  justify  this  assertion?  Most 
would  agree  that  the  following  property  is  essential? 

(i)  That  ths  data  is  (or  may  conceivably  be)  collected  under  a 
repetitive  process,  and  is  collected  without  bias.  Thus, 
sven  if  impractical  to  do  so,  one  could  continue  ths  exper¬ 
iment  indefinitely,  and  the  salsction  of  data  does  not 
depend  on  what  occurs  (that  is,  one  cannot  discard  data 
"after  ths  fact"  as  long  as  such  data  falls  within  our 
‘•apace  of  svents.") 

Some  would  also  insist  that  ths  following  property  must  holdi 

(il)  That  ths  data  must  possess  a  so-callad  "statistical  reg¬ 
ularity"  or  "long-run  stability".  This  property  permits 
one  to  conceive  of  probability  as  a  relative  frequency, 
i.a.,  the  (approximate)  proportion  of  times  that  events 
will  occur  if  ths  experiment  is  continued  long  enough. 

Many  believe  that  justification  of  property  (ii)  is  not  required  in 
order  to  use  the  probabilistic  model.  They  argue  that  experience  shows 
that  satisfying  property  (i)  leads  to  a  satisfaction  of  (ii).  However, 
it  would  not  be  consistent  with  the  purposes  of  this  document  to  enter 
into  this  argument  here. 

d.  Choosing  the  Probability  Punctloq  P.  There  is  usually  insuf¬ 

ficient  time  available  to  use  decision-making  techniques  to  determine 
the  probability  function  [of  a  particular  random  variable)  for  each 
class  of  electronic  equipment  under  consideration.  Fortunately,  there 
is  a  general  agreement  that  the  exponential  density  function  describes 
the  behavior  of  most  electronic  squlpment  reasonably  wall.  This  func¬ 
tion  takss  ths  form  .  ,  where  the  random  variable  t  denotes 

-1_  s-U/S) 

"operating-times-be  tween-failures'"  and  0  is  the  me  an- time -be  tween- 
failures  (MTBF).  By  integrating  this  function  from  t  to  infinity  we 
derive  the  familiar  reliability  function 

R(0  = 


which  gives  the  probability  of  failure-free  operation  for  tlaa  t.  It  is 
also  widely  known  that  in  the  exponential  case,  the  random  variable 
"quantities  of  failures  for  given  test  times"  obeys  a  Poisson  distri¬ 
bution  with  parameter  Oi 

FCX...0  ■ 

x! 


/ 


17 


yields  the  probability  of  exactly  x  failures  occurring  in  test  time  t  if 
the  true  MTBF,  0,  is  known. 

Such  wealth  of  information  concerning  the  math  model  gives  rise  to 
certain  advantages,  not  the  least  of  which  is  simplicity.  (Probably, 
the  most  important  feature  of  a  decision  rule  is  that  it  be  undsrstond 
by  ths  parties  involved.)  Other  advantages,  not  usually  encountered  in 
statistical  literature  (since  mnst  authors  seek  more  general  results) 
are: 

( 1)  Mnre  meaningful  hypotheses  are  established. 

(2)  Producer  and  consumer  risks  may  be  explicitly  quantified 
and  evaluated. 

(3)  Procsdures  may  be  developed  for  comparing  various  accept/ 
reject  criteria,  under  a  given  set  of  conditions,  through  ths  uss  of 
probability  of  acceptance  curves. 

Before  detailing  specific  decision  rules,  it  is  advisable  to  discuss 
some  problems  associated  with  establishing  hypotheses  as  well  as  some 
considerations  which  dictats  how  the  data  is  to  be  interpreted. 

e.  Establishing  Hypotheses  -  The  Interpretive  Process.  It  was  stated 
earlier  that  a  scientific  evaluation  nf  data  is  dependent  upon  the  choice 
of  pertinent  hypotheses  to  compare  the  data  against.  (In  fact,  the 
hypotheses  will  usually  dictate  the  type  of  data  to  collect.)  In  reli¬ 
ability  decision-making,  additional  assumptions  are  required  which  reflect 
the  nature  of  the  equipment  and/or  the  requiremsnts  of  the  producer  and 
consumer,  so  that  rejection  (acceptance)  nf  the  hypotheses  msans  rejec¬ 
tion  (acceptance)  of  the  equipment.  These  assumptions  are  not  a  part  of 
the  mathematical  model,  although  they  are  made  in  consideration  of  it. 

Of  course,  they  must  bs  carefully  scrutinized  prior  tn  the  use  of  any 
decision  rule,  sines  the  final  decision  rests  heavily  upon  these  assump¬ 
tions.  Hence,  ths  main  task  of  reliability  decision-making  is  to  make 
reasonable  assumptions  so  that  hypotheses  may  be  establishsd  which  will 
lsad  to  accept/rsject  decisions  which  satisfy  each  of  the  parties  involved. 
It  shall  be  sssn  that  both  parties  are  largely  concerned  with  the  poss¬ 
ibility  that  incorrect  decisions  will  be  made,  i.e.,  ths  producer  fsars 
that  satisfactory  equipmsnt  may  fail  the  test  while  the  consumer  fears 
that  unsatisfactory  equipmsnt  may  pass  the  test.  Therefore,  an  equitable 
solution  would  be  the  following  interpretive  process:  (l)  properly 
define  11  satisfactory"  and  "unsatisfactory'1;  (2)  develop  quantitative 
expressions  (callsd  risk  functions)  for  the  possibilitiss  of  making  in¬ 
correct  decisions;  and,  finally,  (3)  hold  these  expressions  to  a  minimum 
when  making  accept/reject  decisions  (consistent  of  course  with  other 
practical  constraints).  This  intsrprstivs  tschniqus  is  ussd  in  each 
decision  ruls  presentsd  in  ths  hope  nf  satisfying  both  p&rtiss. 


Specific  .Decision  Rules.  Generally  speaking  there  are  two  approaches 
that  shall  bs  presentsd  callsd  "Fixsd  Tims  Tests"  and  "Sequential  Tests". 
Tneir  similarities  and  differences  cannot  bs  fully  understood  until  ths 
proesdure  of  sach  ruls  Is  known.  Hence,  these  procedures  are  presentsd 
first,  and  ths  next  chaptsr  shall  be  devotsd  to  a  comparative  discussion. 

Tab  Is  I  ( nsxt  pags)  contains  what  may  be  called  ths  "test  logic"  and 
provides  justification  for  the  rules  of  procedure  that  follow  (except  for 
certain  proofs  which  are  placed  in  the  attachments  to  this  pamphlst).  The 
table  should  not  be  read  superficially;  rather,  it  should  be  studisd  cars- 
fully,  one  column  at  a  time.  The  primary  reason  for  placing  this  infor¬ 
mation  in  a  table  Is  to  make  ths  reader  cognizant  of  similarities  and 
differences  of  the  various  tests,  as  e^ch  one  is  studisd.  This  results  in 
a  saving  of  tims  and  effort;  that  is,  having  studisd  and  understood  an 
aspect  of  one  test,  the  readsr  will  nnt,  unwittingly,  dwell  on  this  aspect 
again  for  a  subssqusnt  test.  However,  ths  readsr  whn  is  studying  this 
test  logic  for  ths  first  time,  must  be  on  guard  to  resist  ths  temptation 
nf  studying  two  tssts  s imu 1 tans ou sly .  Ths  ms re  adjacency  of  the  stats- 
msnts  makes  this  tsmptation  ever-present;  and,  if  not  resisted,  could 
result  in  something  lsss  than  full  understanding  of  any  nne  test. 

There  are  many  nther  rulss  that  may  bs  devissd;  here,  we  have  taken  a 
few  of  the  most  popular  onss  and  placed  then  all  within  the  same  frame- 
wotk.  It  is  expected  that  most,  if  not  all,  decision  rules  may  be  fitted 
into  this  framework.  An  obvious  advantage  results  from  such  an  effort, 
namely,  sass  of  communication  betwesn  producer  and  consumer  particularly 
when  a  new  approach  to  dscisl on-making  is  recommended.  A  possible  dis¬ 
advantage,  of  course,  is  that  reasonable  appro&chss  may  be  disapproved 
because  they  do  not  admit  to  this  framework;  however,  it  is  hoped  that 
tnis  disadvantage  is  superseded  by  the  advantage  of  detecting  unreasonable 
approaches  (after  they  have  been  fitted  to  this  framework). 

Certain  details  of  each  nf  these  techniques  -  In  particular,  proofs  - 
nave  besn  placed  in  Attachments  2,  3,  and  4.  Definitions  of  terms  and 
symbols  are  given' in  Attachments  5  and  6. 
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by  direct  application  of  basic  lawa  of  probability  theory  (Attaebnant  l)  • 
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Ter  Cal— a  I  and  H,  equipments  tested  for  length  of  operating  tine  t 

— ■  the  ^Mber  of  -failures  that  occur  in  tine  t,  shall  ba  recorded. 
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the  shore  hypothaala  raqulraa  that  tha  following 

esc— lions  ba  nada: 

1.  Tbt  oon— sr  raqulraa  aqulpnant  with  true  «HP 
fc©*,  and  this  value  my  ba  specified. 

2.  Kqulpneate  with  true  MZBI  ©•  are  ccnslderad 
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3.  The  true  ralde  of  ICHP  is  fixed  by  the  design  See  Colanna  HI  and  IT 

and  a  raids  ©*  is  attainable. 

k.  Thera  ara  only  two  ways  of  mking  an  incorrect 
decision,  whatever  criteria  is  adopted.  These 
are  (l)  Rejecting  "©  't  ©*  is  true"  when  it  is 
aetnally  true,  or  (2)  Rejecting  "0  <  ©*  U  true" 
when  It  is  actually  true. 
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Steps  Which  appear  below  apply  to  all  Coir  me  except  that  for  Colnm  I, 

0  a  QdtO*,  and  far  Colons  I  and  H,  »v»»0  -  1. 

l!  Daflna  "prodnoar  risk" , at ,  aa  follows:  \ 

ba  rejected} s  FJre Jesting  "©  >  ©.  la  true"  when  it  la  true}-  P(R^».\©£^/J 
vbere  ^1*  «  fixed,  known  value,  ©  is  a  fixed,  unknown  value,  and  n^  is 
the  1— bar  of  failures  specified  for  rejection,  (lor  Col—  I  only,  change 
"©£©, "  to"©i  ©•:)  _ 

2.  Define  "cone— r  risk",  /I,  as  follows.  £  «  p{uneaUafactory  Wil— rtvill 
be  acoepted\n  Projecting  "©i©!  la  true"  when  it  la  true*  P{^iM©£  ©i> 
vhera  ©.  la  a  fixed,  known  value,  Q  la  a  flxad,  unknown  value,  and  nt  la 
tha  aupc  of  failures  apaelflad  far  acceptance,  (far  Cal—  X  only,  change 

"®S©L"  to  "©<©*"  and  change  n*  to  n,-l  ) 

3.  Since  ©  Is  unknown,  the  exact  daWnlnatlon  of  <X  and  la  difficult,  How- 
orer.  It  can  bp  shown  ( Attachment  2)  that  ^  and  £  are  bounded  above  by 

D(«.l and  C(n.;©UIt);  that  is,  at  <  ©(n.;©.  jt),  /3  i  C(axi^|t) .  lur- 
ther,  no  bounds  say  be  found  far  these  risks  since  0  remini  un¬ 

known.  Thus,  as  rasing  equality  Instead  of  inequality  gives  us  a  "wewst  case 
ftgwa  tor  tha  risks.  (Tor  Col  I,  change  and  ©g  to  ©•  and  ehanga  a ^  to 
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.  In  addition  to  Coin—  I  and  H,  If  i1+x1f...+  ■  n 

m*  if  we  1st  rt’  *  t,  than  the  ratio 

j  „  U»t?«t!.v.,v«>n»f)  B  _ 

abava  denotes  tha  probability 

of  Innlaw  aicaetlj  ^  (  1  *  l,2,...,r) 
failures  In  aaeh  of  tha  r  m — aaalra  tine  Intervals 
of  length  t  ’ .  (Saa  Attachment  3  for  jroef) . 


Tor  Colxana  XXX  and  XT  equipments  ara  taatad  and  aa  aaoh 
failure  oocurs,  tha  tlaaa  at  vhloh  they  occur  ara  racordad. 


for  Cola—  H,  XXX,  and  XT,  at  aaeh  oocurrenre  of  a  failure  daclda  whether 
©  £©.  or  0  1  0X  idkara  ©#  and  ©*  ara  specified  value*  and  ©#>  ®1#  If  a  decision 
la  not  possible,  oontlana  testing  to  tha  next  fallnra. 


for  Col— a  XX,  XIX,  and  XT,  tha  above  hypothesis  requires  that  tha  following  aaaartlona 

ba  made: 

1.  That  consumer  daalraa  equipment  with  true  MSBP  graatar  than  or  equal  to  0.,  and  this 
value  nay  ba  apaelflad  baforahand. 

2.  that  a  ralna  ©*  (laaa  than  0.)  nay  ba  apaelflad  such  that  valnee  of  true  KfB?  lass 
than  or  aqoal  to  ©^  ara  considered  unsatisfactory. 

3.  That  ralnaa  of  0  such  that  ©x  <  ©  <  Qm  ara  nalthar  satlafaetory  nor  unsatisfactory. 

In  fact,  It  wart  ba  thaeratlcally  assumed  that  such  ralnaa  ara  not  possible  so  that 
rejection  of  0  £  Qm  naans  accept—*  of  0  £  0^. 

*>•  That  tha  true  ralna  of  I GUT  la  flxad  by  tha  dasl^a  and  a  ralna  Q,  la  attainable. 

5-  Thera  ara  only  two  ways  of  asking  an  lncarract  decision,  whatever  criteria  la  adopted. 
That#  ara:  (l)  Rejecting  "0£©#  la  true"  when  it  la  actually  true,  or  (2)'  Rejecting 
"0  <  ©  la  true"  when  It  la  actually  true. 


See  Cola 


In  addition  to  steps  In  Col—  X  and  XX,  we  con¬ 
sider  that  if  R>  1,  then  the  probability  of  obtain¬ 
ing  exactly  n  failures  In  tins  t  la  greater  under 
tha  aenavtlon  that  ©^  la  true  than  It  la  under 
the  asstaptlon  that  0S  la  true;  hence,  ve  tend  to 
ball  era  that  0  ■  ©^.  Slallarly,  If  R<1  wa  tend 
to  ballera  that  0  i©,.  Xn  fact,  It  can  be  shown  that 
tha  true  risks  «.  and /l,  as  defined  above,  will  not 
ba  violated  if  wa  "reject  0  £©#  la  true"  whan 
R  >  and  reject  "0  <  ©^  la  true"  when 

R<  rjL-  . 
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a.  f^xyd  Time  Tests .  Actually,  all  rules  to  be  considered  here  re¬ 
quire  a  fixed  amount  of  testing,  so  that  one  m$y  be  puzzled  by  the  name 
"Fixsd  Time  Test”.  The  reason  for  tnis  name  Is  that  the  rule  specifies 
the  exact  duration  of  tasting  with  no  intermediate  decision  points.  When 
this  amount  of  (operating)  time  has  elapsed^  a  decision  either  to  accept 
or  reject  will  be  made  based  upon  the  number  of  failures  thst  have  occurred. 
It  follows,  then,  that  the  number  of  failures  which  cause  an  accept  de¬ 
cision  oust  be  one  less  than  the  number  of  failures  which  require  a  reject 
decision.  (It  will  be  seen  later  that  Sequential  Tests  operate  much  dif¬ 
ferently).  Definitions  and  explanations  of  symbols  used  to  explain  the 
various  tests  can  be  obtained  by  consulting  Attachments  5  and  6. 

(1)  POISSON  FIXED  TIME  TEST  (PFTT) 

The  rule  of  procedure  (decision  criteria)  is  as  follows: 


Step  I:  Choose 

•(a) 

MTBF  requirement,  say  ©* 

(b) 

Test  time,  t. 

(c) 

Consumer's  risk,  p. 

Step  II j  Using  tables  of  the  Poisson  Distribution  (see  ESDP  80-5) 
find  acceptance  number  x  which  satisfies: 

C(x;0* ; t)  =  p 

NOTE:  In  this  rule,  the  rejection  number  is  x  +  1  and  the  pro¬ 
ducer's  risk,a,is  at  most  1  -  p  since  D(x  +  1  ;©*;t)  =  1  —p 

EXAMPLE:  Let  ©*  =  100,  t  =  170,  and,  p  =  50*,  then  if 

x  =  1 

C(x;0*;t)  =  C  ( 1  ;1 00;1 70) 

=  .497 

=  p  (approximately) 

Hence,  we  may  accept  if  =  Nl70  £  1,  without  violating  the  consumer's 
risk,  p.  Of  course,  in  this  case,  the  producer's  risk,  a,  may  be  as  high 

as 

D(x  +  1  ;©»}t)  =  D(2;1 00;170) 

=  .503 

or  50*  (approximately) 
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NOTE:  This  examp  Is  assumed  that  only  170  hours  were  avail¬ 
able  for  test;  but  if  lOOn  ♦  70  were  available 
(for  n  =  0,  1,  2,  ...)  one  can  choose  x  =  n  and  still 
maintain  risks  of  50*  for  both  parties. 

(2)  PmLlAiternate  Values  of  MTBF) 

The  high  risk  levels  required  by  the  PFTT  where  both  risks  are  eval¬ 
uated  using  one  value  of  0,  can  be  alls via ted  by  basing  such  risks  on 
alternate  values  of  0,  say  0q  and  ©1 .  The  philosophy  behind  this 
maneuver  is  simply  that  it  does  not  sssm  reasonable  that  the  consumer 
requires  MTBF  =  0*  =  0q  and  values  slightly  below  0  are  deemed  un¬ 
satisfactory.  It  is  more  likely  that  a  value  of  MT§F  exists  which  is 
le^g  than  ©Q  say  0^ ,  such  that  if  0  £  0^,  operational  effectiveness 
would  seriously  deteriorate.  Therefore,  this  lower  value  0  is  de¬ 
termined  in  advance  and  ths  consumer's  risk,  p,  is  evaluate!  using  0 
while  the  producer's  risk,  a,  is  computed  using  the  value  0  .  This  1 
has  the  effect  of  establishing  a  "zone  of  indifference'',  namely,  those 
values  of  0  between  ©1  and  0Q.  Although  valuss  of  MTBF  falling  within 
this  zone  are  not  deemed  ’'satisfactory'*  neithsr  are  they  deemed  "unsat¬ 
isfactory",  and  the  probability  of  accepting  such  equipments  is  per¬ 
mitted  to  rise  higher  than  the  predetermined  risk  level,  p.  Also,  the 
probability  of  rejecting  such  equipments  is  permitted  to  rise  higher 
than  the  predetermined  risk  level,  a.  A  special  feature  of  this  tech¬ 
nique  is  that  for  given  values  of  a  and  p,  as  the  size  of  the  zone  of 
indifference  increases,  the  required  test  time  t  decreases.  Much  of 
what  follows  is  a  repetition  of  the  PFTT  procedure  with  certain  modi¬ 
fications. 

Step  It  Choose 

(a)  MTBF  upper  and  lower  bound  values,  and  ©^ . 

(b)  Test  time,  t. 

(c)  Consumer's  risk,  p. 

Step  II:  Using  tables  of  the  Poisson  Distribution  (see  ESDP 
80-51  find  acceptance  number  x  which  satisfies 

C(x;©1;t)  =  p 

NOTE:  In  this  rule,  the  rejection  number  is  x  +  1,  but  the 
producer's  risk,  a  is  not  as  high  as  1  -  p  since 
D(x  +  1;0Q;t);  that  is,  f  is  computed  using  ©^  and 
the  "worst  case"  figure  for  a  is  computed  using  ©Q. 

EXAMPLE:  Let  0q  =  200,  ©1  =  100,  t  =  2,000,  p  =  10*,  then  x  =  U  yields: 

C(xf©.j;t)  =  C(  14; 100; 2,000)  =  .105  -  p.  (approximately) 
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Heno*,  v*  m*j  *ewpt  if  »t  *  »,  ««  =  14  without  Tiolating  the  con- 
•umer'  •  ri»k,  p.  In  thi»tCMe7'lB8  producer' »  risk,.  s>  Is  »t  most 

D(x  +  1  ;©0i  t) 

=  D(I5;200;2,000) 

.=  .063, 
or  8%  approximately. 

b.  Sequential  feats.  In  fixed-time  tests  an  accept  deciaion  ia  not 
poaaible  until  the  prescribed  teat  time  ha a  been  completed.  In  the  laat 
example,  for  inetance,  the  deciaion  waa  to  accept  if  14  or  leea  failures 
occurred  in  2,000  hour a  of  testing.  low,  suppose  that  after  1,000  houre 
of  testing  fero  failures  had  occurred.  One  might  suspect  that  the  con¬ 
sumer  could  acoept  at  this  point  with  a  low  riek.  The  riek,  in  fact, 
would  be  fc(0;100j  1,000)  =  .000045  (less  than  one  ten- thousandths  of  one 
percent!)  Considerations  such  as  these  lead  to  the  concept  of  "sequen¬ 
tial  testing.-  This  concept  can  be  stated  eimply  as  follows;  Develop 
procedures  which  would  permit  decisions  at  such  (earlier)  times  without 
violating  the  pre-selected  risk  levels. 

Two  types  of  sequential  tests  shall  be  presented.  These  are  called 
(l)  Probability  Ratio  Sequential  Test  (FRST)1  and  (2)  Poisson  Sequen¬ 
tial  Teat  (PST), 


(1)  PROBABILITY  RATIO  SEQUENTIAL  TEST  (FRST). 

The  rule  of  procedure  (decision  criteria)  is  as  follows: 

Step  (l)»  Chooee 

(a)  MTBF  upper  and  Lower  bound  values,  0Q  and  0^ , 

(b)  Consumers  risk,  p. 

(c)  Producer's  risk,  a. 


1  For  a  more  general  discussion  of  the  FRST  see  Wald's  "Sequential 
Analysis"  published  by  John  Vlley  It  Sons,  Chapman  and  Hall  Ltd. 
London  (1947).  See  also,  Attachment  3. 
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Step  (2):  For  successive  numbers  of  failures  i  ~  0, 1,2, . . . ,e to. , 


‘i  "(r?v)  [“  (rr,v  ‘"‘"’l 


[“  (4*) -)<-.>! 

where  the  derived  t^'s  repreeent  minimum  test  times  for  acceptance 
if  i  or  less  failures  havs  occurred,  and  the  derived  t^'s  represent 
maximum  test  times  for  rejection  If  j  or  more  failures  have  occurred. 

(See  Attachment  3  lor  derivation  of  formulas  for  t^  and  t.,  above). 

Step  (3):  Truncation.  To  prevent  this  test  from  continuing  to 
undesirable  lengthe,  the  fallowing  truncation  procedure  may  be  used.* 
Scanning  tables  of  the  Poieson  Distribution  find  a  number  x  such  that 

C(x}0,»t) 

and  D(x;0Q;t)  =  a. 

This  is  always  possible  because  of  the  fact  that  the  formulas  for  t^  and 

t.,  given  in  Step  (2),  yield  accept/reject  times  which  hold  the  true 
risks  somewhat  below  a  and  p.1 

In  Chapter  IV  a  specific  test  ehall  be  derived  using  the  PRST  tech¬ 
nique  and  an  analysis  of  the  actual  risks  shall  be  made.  It  will  become 
apparent  that  the  PRST  holds  the  risks  lower  than  originally  intended. 

This  is  the  result  of  using  the  approximations  1  -  B  and  B  for 

constants  A  and  B,  and  the  effect  is  that  a  longer11  test  duritloft  is  re¬ 
quired.  In  fact,  if  MTBFs  are  high  (say,  in  the  order  of  500  hours  or 
more)  the  cost  of  testing  and  the  consequences  of  delayed  decisions  makee 
the  PRST  techniques  prohibitive.  This  is  especially  eo  if  the  true  MTBF 
of  the  equipment  falls  in  the  zone  of  indifference  (between  0^  and  0Q). 

The  following  technique  attempts  to  combine  certain  advantages  of  the 
PFTT  and  the  PRST  while  removing  certain  dieadvantages  in  order  to  cope 
with  the  above  stated  situation.  These  advantages  and  dieadvantagee  shall 
be  discussed  in  Chapter  IV. 

1  Justification  for  this  method  need,  not  be  explained  (again)  because 
of  its  similarity  with  the  PFTT  (Alternate  Values  of  MTBF). 
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(2)  POISSON  SEQUENTIAL  TEST  (PST). 

The  rule  of  procedure  (decision  criteria)  is  as  follows! 

Step  (i)l  Choose 

(a)  KTBF  upper  and  lower  bound  values,  0^  and  0^ . 

(b)  Consumer^  risk;,  p. 

(c)  Producer*  s  risk,  o. 


Step  (2):  Using  tables  of  ths  Poisson  Distribution  starting  with 
bmaii  values  of  U^  ,  find  the  first  point  at  which 

C ( i ; 01 ; t^ )  =  0  (for  i  =  0, i,2, . . . , e tc) 

That  is,  derive  UQ  =  tQ^  that  corresponds  with  <5(0^  |tQ)  =  p,  then 

=  t^yg  that  corresponds  with  C ( 1 ; 0^ ;  t^ )  =  p,  and  so  on.  Then  solve 

each  of  the  aquations  =  t^yg  ^or  ^i  , . . .  ,etc. , 

where  each  t  represents  minimum  allowable  test  time  for  acceptance 
if  i  or  less^faiiures  have  occurred  in  that  time. 


Step  (3):  Follow  exactly  ths  same  procedure  as  Step  (2)  using 
a,  0q,  and  D(x;0Q}t^)  to  derivs  t^'s  where  each  t^  is  maximum  allowable 

test  tims  for  rejection  if  j  or  more  failures  have  occurred  by  that 
time. 


Step  (4) l  Truncation.  This  test  will  terminate  itself  automati¬ 
cally,  i.e.,  eventually  a  point  will  be  reached  where  t^  =  t^  and 
i  =  j  -  1  (the  number  of  failures  for  an  accept  decision  is  one  less 
than  ths  number  of  failures  for  a  reject  decision)  However,  the  test 
may  be  truncated  much  sarlisr  as  a  result  of  considerations  which 
are  given  in  Attachment  4  ,  to  this  pamphlet. 
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Chapter  4 


COMPARATIVE  ANALYSIS  -  CONSTRUCTION  AND  APPLICATION 
OF  PROBABILITY  OF  ACCEPTANCE  CURVES 


1.  The  Basic  Too  let 

Each  of  the  preceding  dscision  rules  employed  the  probabilistic 
model  using  ths  Poisson  frequency  function  as  the  probability  function 
P,  that  iai 


p/  q.\  _  -U  .x  _  (The  probability  of  sxactly  x  failures) 

x*  *  * - ”  ^occurring  when  U  =  t/0  is  known  * 


xl 
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where  t  is  the  operating  time  and  0  is  the  trus  MTBF  of  the  equipment. 
The  following  formulae  may  be  derived  from  P(x;0;t)  using  basic  axioms 
of  probability  theory  (see  At  tactertant  ;1 ) : 


C(x;0;t)  =  S—  Vr  = 


-U  ..r  _  fahe  probability  of  x  of  fswer  faiiuresT 


rj  ^occuwingwhen  U  =  t/0  is  known 


D(*,Q;t)  -f  *  f(Xl)|if  x  =  o) 


j^The  probability  of  x  or') 

=  /more  failures  occurring  v 
\yrhen  U  =  t/0  is  known  ) 


To  determine  the  probability  of  faiiure-frss  operation  for  time  t,  it 
suffices  to  calculate 

P(0;0j t)  =  e-t/0  M  =  s'1/8 

01 

which  is  considered  to  be  the  reliability  function,  R(t).  Thus  know¬ 
ledge  of  0  is  equivalent  to  knowledge  of  the  equipment  reliability  (for 
given  test  times.) 

A  technique  may  now  be  given  for  analyzing  the  differences  in  these 
decision  rules.  Before  doing  so,  however,  it  is  convenient  to  discuss 
the  features  of  Fixed  Time  Tests  as  opposed  to  Sequential  Tests. 


2.  Fixed  Time  Te^ts  versus  Ssauentlal  Tests.  For  ease  of  discussion, 
the  following  comments  are  directed  towards  comparing  the  PFTT  (Alternate 
values  of  MTBF)  and  the  PST,  sines  these  rules  differ  only  in  that  the 
former  assumes  a  "fixed  tims"  approach  whereas  the  latter  assumes  a 
"sequent iai"  approach.  The  fixed  time  approach  considers  that  the  amount 
of  time  t  that  may  be  allotted  for  reliability  testing  is  extremely 
limited  so  that  achieving  more  than  one  failure  in  this  time  is  rather 


jHNMMMMi 
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unlikely.  Hence,  impute  qf  0  ,  0  ,  fixed  time  t,  producer  and  consumer 
risks  are  combined  to  determine  maximum  quantities  of  failures  for 
acceptance  and  minimum  quantities  of  failures  for  rejection  (which, 
naturally,  differ  by  one),  in  the  given  time  t.  Diagrams ti cal ly 


PUBIS _ 

*o*0j  »a,P»t 


_ OUTPUTS _ 

Accept  if  §  failures  £  x 

JJL#  failures  >  x  1 1 


Occasionally  risks  may  require  adjustment  (upwards)  because  the  test 
time  is  so  limited. 

On  the  other  hand,  in  ths  sequential  approach,  test  time  is  more 
flexible  (though  not  any  lass  important).  Thus,  inputs  of  0^0  ,  pro¬ 
ducer  and  consumer  risks  are  combined  with  successivs  numbers  or  failures 
0,  1,  2,  and  so  on,  to  determine  minimum  test  times  for  acceptance  and 
maximum  test  times  for  rejection  to  be  associated  with  each  of  these 
numbers  of  fai lures I 


am?  . 

©»°i 

consecutive  #  of 


(Other  procedures  are  used  to  find  a  point  for  truncation  without 
violating  the  prescribed  risks.)  Occasionally,  when  high  MTBF' &  are 
involved,  the  amount  of  total  test  time  becomes  unreasonable  unless  the 
risks  are  adjusted  (upwards).  However,  this  adjustment  is  not  as  severe 
as  in  the  case  of  the  fixed  time  approach. 

Essentially,  than,  ths  sequential  approach  is  merely  a  series  of 
fixed  time  tests  except  that  certain  test  times  may  have  acceptance 
numbers  only  and  other  test  times  may  have  rejection  numbers  only.  The 
test  ends,  of  course,  when  an  acceptance  time  coincides  with  a  rejection 
time  and  the  quantity  of  failures  permitted  at  that  acceptance  time  is 
one  less  than  that  permitted  at  the  rejection  time. 

Consequently,  there  is  little  need  to  discuss  the  conditions  which 
make  the  fixed  time  test  preferable  to  the  sequential  test.  The  answer 
is l  Use  the  sequential  test  whenever  ths  allotted  test  time  permits; 
when  time  is  insufficient,  resort  to  the  fixed  time  test. 

3.  PRST  versus  PST.  Let  us  now  assume  that  time  is  rather  flexible 
(but  by  no  means  unlimited!).  Our  problem  then  reduces  to  choosing  be¬ 
tween  ths  PRST  and  PST.  It  shall  turn  out  that  the  key  consideration  is 
again  that  of  the  amount  of  time  that  may  reasonably  be  allotted  to 


OUTPUTS _ 

Test  times  for  Accept 
Test  times  for  Reject 


reliability  testing.  To  see  this  more  clearly,  suppose  that  the 
following  conditions  applyt 


i 


0Q  *  600  hours  =  Satisfactory  value  of  MTBF 

®1  =  400  hours  =  Unsatisfactory  value  of  MTBF 

a  =  20£  =  F^  Satis  factory  equipment  will  be  rejected} 

P  =  *  P  {  Unsatisfactory  equipment  will  be  accepted} 

Use  of  PRST  technique  as  given  in  paragraph  2  of  Chapter  3  yields  the 
following  acoept/reject  criteria! 


mm 

Risk  I*val  £  20£  Discrimination  Ratio*  3/2  =  ^o/O^ 

Total  test  time* 


>tai  observed 

Reject 

Accept 

failures 

(Equal  or  less) 

(Equal  or  more) 

0 

N/A 

2.8 

1 

N/A 

3.6 

2 

N/A 

4.4 

3 

N/A 

5.2 

4 

0.5 

6.0 

5 

1.3 

6.8 

6 

2.1 

7.6 

7 

2.9 

8.5 

• 

3.7 

9.3 

9 

4.5 

10.1 

10 

5.3 

10.9 

n 

6.1 

11.7 

12 

6.9 

12.5 

13 

7.7 

13.3 

14 

8.6 

14.1 

13 

9.4 

14*6 

16 

10.2 

14.6 

17 

.  11.0 

14.6 

18 

11.8 

14*6 

19 

14.6 

N/A 

•Total  test  time  is  expressed  in  multiples  of  6q 

There  is  nothing  to  stop  equipments  from  remaining  in  the  continue 
test  region  for  the  entire  span  of  the  test.  In  other  words,  there  is 
a  distinct  possibility  of  the  test  lasting  14.6  multiples  of  0  or 
14.6  x  600  *  8,760  hours.  Even  if  two  models  were  simultaneously  tested 


with  their  teat  time*  end  quantities  of  failures  combined,  this  would 
require  4,330  test  hours  or  more  than  6  months  of  24-hour-a-day  relia¬ 
bility  testing! 


Now  consider  use  of  the  PST  technique  as  given  in  Chapter  3.  Fol¬ 
lowing  that  procedure  yields  the  following  accept/reject  criteria: 

PST  ACCEPT /REJECT  CRITERIA 


Risk  Level  =  20* 
Total  observed 

fftUma - 


Discrimination  Ratio:  3/2  = 


1 


Total  test  time* 
for  REJECT 
(equal  or  less) 


Total  test  time* 
for  ACCEPT 
(equal  or  more) 


0  N/A  1.60 

1  N/A  3.00 

2  N/A  3v4£ 

3  2.30  3*45 

4  3.45  N/A 


•Total  test  time  is  expressed  in  multiples  of  ©^ . 

Here,  maximum  test  time  is  3*45  multiples  of  ©  or  (3.45)  x  400  =  1,3^0. 
If,  again,  two  models  could  be  tested  simultaneously,  the  number  of  test 
hours  required  would  be  (at  most)  690,  or  less  than  one  monthl 

A  reasonable  question,  at  this  point,  is:  How  can  both  the  PRST 
and  the  PST  criteria  be  based  on  the  same  risks  (  a  =  %  =  20*)  and  the 

©  /  600/  3/ 

same  discrimination  ratio  (d  =  °'  1  =  '400  =  2  )  and  yet  yield 

such  differing  criteria?  Part  of  the  answer  is  simply  that  the  PRST 
does  not  use  the  risk  functions 

« =  PK  *  ni!  6  =  eo> 

»•  p(NtW  e=  ei> 

directly  to  derive  the  accept/rejsct  criteria,  whereas  the  PST  does. 

Hence,  the  PST  allows  a  and  p  to  reach  these  levels,  but  the  PRST  keeps  the 
risks  well  below  these  levels.  The  rest  of  the  answer  is  supplied  by  the 
fact  that  the  PRST  does  not  use  previously  gained  information  in  order  to 
find  a  point  for  truncation  whereas  the  PST  does. 

Another  question  that  may  be  raised  is:  Why,  then,  do  we  need  the 
PRST— why  not  simply  utili2e  the  PST  as  the  decision  rule,  since  it  makes 
decisions  which  are  consistent  with  the  predetermined  risk  levels  and 
does  so  In  less  time? 


'-'U; 
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The  answer  is  furnished  by  looking  at  "probability  of  acceptance 
curves”  for  each  of  these  tests.  Such  curves  hre  constructed  as 
follows: 

At  any  decision  point  of  the  PRST  or  PST,  one  may  plot,  as  a 
function  of  0,  the  following: 

P^a  or  less  failures  occur  in  time  t*  | 

-  ±  e-t»/0(f/8) r 

r=D  r! 

where  a  is  the  number  of  failures  specified  for 
point  t*.  Evidently,  this  expression  gives  the 
ance  as  a  function  of  the  true  MTBF  0.  Looking 
point  of  the  PRST  previously  derived  gives 


PRST  PA  CURVE  (at  2.8  multiples  of  ©q) 
-  0  Fai lures  - 


This  curve  shows  quite  clearly  that  the  PRST  holds  the  risks  much 
lower  than  origins lly  intended  at  earlier  decision  points. 


e) 


acceptance  at  decision 
probability  of  accept- 
at  the  first  decision 


.  . 


-  ss&iT&g  . 


Plotting  this  sAffiA  function  for  the  first  decision  point  of  the 
PST  which  was  derived  givesi 


PST  PA  CURVE  (at  1.6  mult.) 
-  0  7 allures  - 


Note  that  for  values  of  MTBP  less  than  or  equal  to  400  hours,  the 
probability  of  acceptance  is  below  20%.  But  there  is  a  significant 
area  under  the  curve  to  the  left  of  the  400  hour  point,  whereas  the 
PRST  has  not.  Xt  is  clear  then,  that  the  PRST  provides  better  pro¬ 
tection  against  accepting  unsatisfactory  equipments,  since  a  similar 
situation  holds  at  any  decision  point  that  is  plotted. 

4.  PA  Curves  for  Pixed  Time  Tests.  Now  suppose  that  the  MTBP,  the 
number  of  models  available  for  test,  and  operational  commitments  are 
such  that  even  the  PST  provides  too  long  a  test.  Por  the  case  pre¬ 
viously  considered,  suppose  that  only  one  model  was  available,  causing 
possible  test  duration  of  1,380  hours  (about  two  months)  which  is  too 
long.  Suppose  further  that  only  600  hours  of  testing  could  be  permitted. 
Again  PA  curves  could  be  used  to  choose  the  most  appropriate  Pixed  Time 
decision  rule.  Por  example,  if  0  failures  is  specified  as  acceptance 
number  (here,  \  failure  is  the  rejection  number)  the  PA  curve  taxes  the 
form 


It  should  be  observed  that  since  the  rejection  number  is  one  more  than 
the  acceptance  number,  the  probability  of  rejection  is  "one  minus  the 
probability  of  acceptance”.  (This  does  not  hold  for  sequential  tests, 
however.)  Thus,  this  curve  shows  that  if  the  true  MTBP  is  800  hours  (200 
more  than  required),  there  is  better  than  a  50%  probability  of  rejection, 
since  there  is  only  a  47%  chance  of  acceptance. 

Changing  the  acceptance  number  to  1  (the  rejection  number  to  2)  pro¬ 
duces  a  rather  striking  change  to  the  PA  Curve,  as  shown  below: 


Here,  if  the  true  MTBP  is  800  hours  there  is  an  82%  chance  of  accept¬ 
ance.  But  now,  there  is  a  40%  chance  that  equipment  with  300  hours  MTBF 
(300  less  than  required)  would  pass  the  test.  Clearly  one  must  choose' 
either  0  or  1  as  the  acceptance  number,  and  this  Is  a  formidable  task, 
since  there  is  considerable  difficulty  in  holding  both  Air  Porce  and  pro¬ 
ducer  risks  at  low  levels. 
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5.  Advantages  and  Disadvantages.  As  a  result  of  analyzing  PA  curves  for 
the  three  kinds  of  tests,  it  is  possible  to  summarize  certaia  conclusions 
in  terms  of  advantages  and  disadvantages  of  using  each  of  these  techniques. 
These  are  tabulated  on  the  following  pages. 
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PFTT 

APYAWAQEa 


PRST 

APVWIAffis 


PST 

wmm®. 


1.  Teat  ia  simple  to  admin¬ 
ister — squat iona  ar«  ava liable 
Which  permit  specification  of 
the  accept/reject  criteria  in 
terms  of  MTBF  requirements. 

2.  Permits  specification  of 
risks  before  testing  begins. 

3.  The  exact  test  time  is 
known  in  advance  and  may  be 
scheduled  and  costed. 

4.  Possible  to  devise 
criterie  to  fit  most  any 
IOC  dete. 

5.  If  producer  has  designed 
equipment  with  MTBF  ouch 
higher  than  required,  he  may 
be  willing  to  assume  much 
higher  risk,  thus  shortening 
test  time  coneiderebly . 


1 .  Permits  earlier  decisions  for 
equipments  extremely  better  or 
extremely  worse  than  required. 

2.  The  test  is  simple  to  admin¬ 
ister— equations  are  evsileble 
which  permit  specificetion  of  the 
eccept/reject  criterie  in  terms  of 
MTBF  requirements. 

3.  Hes  intuitive  appeal  es  e 
result  of  using  the  "Likelihood 
fraction"  (in  addition  to  risk 
levels)  as  e  basis  for  deriving 
accept  /Reject  criteria. 

4.  permits  specificetion  of  risks 
(consumer’s  and  producer's)  before 
testing  begins.  (However,  for 
risks  above  certain  levels,  the 
epproximat ions  used  to  derive  the 
PRST  are  no  lonper  valid.) 


1 .  Test  is  simple  to  administer —equa¬ 
tions  ere  evsileble  which  permit  speci¬ 
fics  tion  of  the  eccept/reject  criterie 
in  terms  of  MTBF  requirements. 

2.  Permits  earlier  decisions  for  satis¬ 
factory  or  unsatisfactory  equipments. 

3.  Permits  earlier  truncation  of  tasting 
es  e  result  of  utilising  the  fact  that 
the  equipment  was  not  accepted  or  rejected 
et  previous  decision  points. 

4.  Permits  specification  of  risks  before 
testing  begins. 

5.  Possible  to  devise  criteria  to  fit 

most  eny  IX  date.  (However,  if  test  t 

duration  is  shortened  too  much,  the 
PST  reduces  to  the  PFTT.) 

6.  If  producer  has  designed  equipment 
with  MTBF  much  higher  than  required, 

he  may  be  willing  to  assume  much  higher 
risk  thus  shortening  test  time 
considerably. 

7.  Although  the  exact  test  time  is 
not  known  in  advance,  the  entire  span 
of  the  test  is  short  (as  compared  with 
PRST),  and  good  approximations  of  last 
duration  can  be  made  for  purposes  of 
costing  and  scheduling  the  taat. 


PFTT 

DISADVANTAGES 


.  PRST 

dxsapya:  X3. 


PST 


1.  Hss  little  intuitive  eppeal 
since  tl>e  accept/reject  criterie 
la  based  solely  upon  the  risks 
involved . 

2.  If  only  one  value  of  MTBF  is 
speoified,  one  of  the  risks  is 
the  complement  of  the  other;  that 
is,  if  the  consumer's  risk  is 
10%,  the  producer’s  risk  is  90%. 
Thus,  one  party  or  both  parties 
must  assume  e  high  degree  of 
risk.  (This  can  be  partially 
alleviated  by  specifying  upper 
and  lover  bound  values.) 


1 .  The  test  logic  is  concerned  with 
testing  one  value  of  MTBF  against 
another.  Thus,  a  discrimination  ratio 
must  be  selected;  that  ic,  MTBF  upper 
and  lower  bound  values  must  be 
decided  upon. 

2.  Requires  exoesaive  test  time  for 
equipments  whose  true  MTBF  fslla 
within  the  "zone  of  indifference" 
(values  between  the  upper  and  lower 
bound  values ) ,  although  such  equip¬ 
ments  are  not  deemed  unsetisfectory. 


1.  Hss  little  intuitive  eppasl  since 
the  eccept/reject  criteria  is  based 
solely  upon  the  risks  involved. 

2.  The  test  logic  is  concerned  with 
testing  one  value  of  MTBF  against 
another.  Thus,  e  discrimination  ratio 
must  be  selected;  that  is,  MTBF  upper 
and  lower  bound  valuer  must  be 
decided  upon. 


3.  The  true  risks  (consumer's  end  pro¬ 
ducer's)  usually  are  held  much  lover 
than  those  agreed  upon  before  testing 
began,  thereby  causing  excessive  test 
time.  When  risks  of  10%  or  more  are 
initially  prescribed,  the  test  holds 
the  risks  et  7%  or  more  below  these 
values. 

4.  Formulas  exist  for  evaluating  ex¬ 
pected  test  time,  but  es  e  function  of 
the  true  MTBF  which  is  unknown;  hence, 
scheduling  and  costing  this  test  is 
difficult.  In  certain  cases,  fear  of 
having  to  run  the  entire  span  may 
ceuae  the  teat  to  be  waived. 

5.  Extremely  difficult  to  deviee  testa 
to  fit  early  IOC  dates. 

6.  The  procedure  for  truncation  does 
not  utilize  information  concerning 
the  performance  of  equipment  at 
previous  decision  points,  which  results 
In  unnecessary  testing. 


Chapter  5 

SUmARY  AND  CONCLUSIONS 


1.  The  preceding  chapters  have  attempted  to  provide  a  framework;  for 
various  Reliability  decision-making  techniques  in  order  to  develop  a 
rationale  for  choosing  appropriate  statistical  accept/reject  criteria. 
Specifically  the  foliowing  points  were  emphasized. 

a.  That  no  one  decision  rule  can  be  applied  to  all  procurements. 

b.  That  meaningful  requirements  cannot  be  established  without 
consideration  of  the  available  methods  (decision  rules)  for  assuring 
that  such  requirements  are  satisfied.  For  example,  the  sequential 
test  methods  that  were  presented  here  required  that  MIBF  upper  and 
lower  bound  values  be  stipulated. 

c.  That  any  decision  rule  carries  with  it  certain  risks  for  both 
consumer  and  producer  as  a  result  of  the  fact  that  only  a  small  portion 
of  the  time  domain  (usefui  life)  will  be  sampled  during  the  demonstration. 
Thus,  risk  levels  must  be  decided  upon  before  the  demonstration  begins. 

d.  That  risk  levels  cannot  be  chosen  without  considering  certain 
aspects  of  the  procurement  situation  such  as  schedules,  cost  of  testing,* 
etc.,  all  of  which  affect  the  amount  of  time  that  may  be  devoted  to. 
reliability  testing.  Neither  should  a  “time  allotment"  determination 

be  made  without  considering  the  effect  upon  risk  levels. 

e.  That  if  sufficient  time  is  allotted,  "sequential  tests"  are 
preferred  over  "fixed  time  tests-  because  they  make  possible  eariier 
decisions  without  violating  the  prescribed  risks. 

2.  An  important  and  strongly  related  topic  was  avoided  in  this  dis¬ 
cussion,  namely,  the  meaning  (consequences)  of  accept-re ject  decisions. 
Besides  the  fact  that  such  considerations  go  beyond  what  we  set  out  to 
accomplish,  such  related  topics  as  "incentives" ,  "penalties",  etc., 
would  be  involved,  and  would  probably  serve  to  distract  the  reader  from 
our  intended  purposes.  Moreover,  it  is  expected  that  other  documents 
will  give  adequate  coverage  to  this  important  topic. 
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BASIC  LAWS  0?_ggptf  am 


1.  Underlying  Assumption.  The  elements  of  set  theory,  wherein  e  point 
(element)  ia  called  an  "outcome  of  an  experiment"’ ,  the  coliection  of  all 
points  (outcomes)  Is  called  the  ’’sample  space"  (and  denoted  by  G),  and 
any  sub- col  lection  of  points  (outcomes)  of  G  is  called  an  "event".  In 
particular  G  is  an  event,  and  the  collection  consisting  of  no  points 
(outcomes)  Is  an  event  (called  the  "null  event"  and  denoted  by*0).  A 
"random  variable"  is  a  numerical  -  valued  function  defined  over  the 
sample  space  G;  l.e.,  a  rule  which  assigns  exactly  one  number  to  each 
outcome.  We  write"u£ G"vhen  we  mean  "w  is  an  element  of  G." 


2.  Basic  Axioms.  We  assume  the  existence  of  a  function  P  satisfying 
the  following  axioms'. 

a.  Axiom  Is  G }  =1 

b.  Axiom  2:  p{0  }  =0 

c.  Axiom  3s  0  1  £  1  for  any  event  E 

d.  Axiom  4.  P{E10E2>=  p{  E2}  -  P^hE^ 


NOTE:  If^AE^  =  {  0}  then 

p{e1Ue2}=  p{El\  +  p{e2\ 


3.  Cumulative  Distribution  Function.  For  any  random  variable  1  =  1  (w) 
the  function  defined  by 

1  *1  ^  =  P  ^X(w)  i  a}  where  (-~<  a  <»  )  and  w£G>  is 

called  the  cumulative  distribution  function. 


4.  Properties  of  the  Cumulative  Distribution  Function.  Tne  cumulative 
distribution  function  has  the  following  basic  properties: 

a.  It  is  a  non -decreasing  function. 

b.  F^  (--)  =  0 

c.  Tx  (  -)  =  1 


P^E^is  notation  for  "The  (numerical)  probability  of  the  event  E  occur¬ 
ring4  .  j  E^  f\  signifies  the  simultaneous  occurrence  of  E^  and  E,  . 

^  U  eJj  is  an  event  which  occurs  when  eLther  or  E  occurs. 
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DERIVATION  OF  POISSON  FUGSD  TIME  TEST  (PFTT) 
AND  POISSON  SEQUENTIAL  TEST  (PST) 


1.  PFTT i 


a.  MlfllUop  9f  flfld  pr9dv<^r  Leb  the  producer 

risk,  a,  be  defined  by 


a  =  P^satisf actory  equipment  will  be  rejected} 


and  let  the  consumer  risk,  f,  be  defined  by 


p  =  P^unsatlsfactory  equipment  will  be  accepted} 


where  "satisfactory  equipment"  Is  defined  as  equipment  with  true  MTBF 
greater  than  or  equal  to  0*,  and  "unsatisfactory  equipment"  Is  defined 
as  equipment  with  true  MTBF  less  than  0*. 


b.  QuaatiXlc&Ugfl  of  Marnmr  AQd  grrturer  rlsta*  if  equipments 
are  tested  for  length  of  operating  time  t,  and  if  accept/reject  decisions 
are  made  based  on  N. ,  the  cumber  of  failures  that  occur  in  time  t,  then 
assuming  that  ■  failures  occur,  the  quantity 

P{nt  i  m  \  9  *  cl 

gives  the  producer's  risk,  a,  if  a  reject  decision  is  made;  and,  the 
quantity 

p{»t  £  •  I « <  *} 

gives  the  Consumer4  s  risk,  if  an  accept  decision  is  made.  In  these 
expressions,  0*  is  assumed  fixed  and  known,  whereas  0  Is  assumed  fixed  but 
unknown. 


0.  Evaluation  of  consumer  and  producer  risks t  In  spite  of  having 
developed  expressions  (in  paragraph  b,  above)  which  seemingly  quantify  the 
risks,  exact  evaluation  of  these  expressions  is  impossible  because  0,  the 
true  MTBF,  remains  unknown.  (Ve  have  no  desire  to  treat  0  as  a  random 
variable).  However,  if  we  assume  that  the  random  variable  "quantities  of 
failures  for  given  test  times"  obeys  a  Poisson  distribution  with  parameter 


that  is. 

C(x;Q;t)  = 

-0  „r 

:  Z_  & _ 1L  = 

p(n  £  x, 

,  if  U  =  t/o  Is  known} 

r=0  .cl 

\  1 

D(x;8jt)  : 

=  A  -  C^,  If  x  i  l' 

)■  K”. 

£  x,  if  U  =  t/Q  is  known} 

\  1  .  1  f  «  =  0, 

then  upper  bounds  may  be  found  for  a  and  f.  This  is  accomplished  as  follows: 
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Consider  the  expressions 
=  Plub{Nt*“l8‘} 

r  =  piub{Nt^“le-} 

where  0  and  0  denote  that  0  varies,  (not  randomly),  but  0  means  that  0 
is  restricted  to  values  greater  than  or  equal  to  0*  and  0  means  that  Q 
is  restricted  to  values  less  than  0*.  (p„  ^  denotes  the  least  upper  bound 

of  the  probabilities  obtained  as  0  assumes  any  of  the  values  indicated). 
Obviously  a  and  p,  us  defined  in  paragraph  b,  above,  are  less  than  or 
equal  to  a'  and  J',  respectively,  (simply  by  the  definition  of  " least 
upper  bound”. )  Also,  by  scanning  tables  of  the  Poisson  distribution  (see 
ESDP  80-5)  it  is  obvious  that  for  fixed  x  and  t,  as  0  decreases.  C(x;0;t) 
decreases,  and  as  0  increases.  D(x;0;t)  decreases.  Thus  we  nave  shown 
that 

a  £  a'  =  D(m;0*;t),  if  a  reject  decision  is  made  on  tr.e  Oasis  of  ’*• 
P  £  =  C(m;0*;t),  if  an  accept  decision  is  made  on  the  basis  of  li. 


In  words,  we  have  shown  that  evaluating  D(m;0;t)  using  0=0*  gives  a 
"worst  case"  probability  for  the  producer's  risk,  a.  Similarly  C(m;0*;t) 
is  a  "worst  case"  probability  for  the  consumer’s  risk,  p.  But  no  smaller 
bounds  may  be  found  for  a  and  ^  since  0  remains  unknown. 

d.  The  PFTT  Technique:  See  page  22  of  this  document. 


1.  The  "least  upper  bound"  of  a  set  of  numbers  is  »  .  ,.u  L  . 

greater  than  or  equal  to  any  number  in  the  set.  (It  may  or  may  not 
belong  to  the  set). 
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2.  PFTT  (Alternate  values  of  M1BF)  and  PST. 

a.  Same  as  l.a,  above,  except  that  "satisfactory  equipment"  is 
defined  as  equipment  with  true  MTBF  greater  than  or  equal  to  0  and 
"unsatisfactory  equipment"  is  defined  as  equipment  with  true  MIBF 
less  than  or  equal  to  0^ ,  where  0^  <  0q. 

b.  Same  as  l.b,  above,  except  that  the  producer's  risk,  a,  is 
given  by  ( 

PK  *  m|e  *  eo 

and  the  consumer's  risk,£,  is  given  by 

p{nt  i  o|e  £  a,} 

c.  Same  as  l.c,  above,  except  that  0  means  that  0  is  restricted 

to  values  greater  than  or  equal  to  0  and  0  means  that  0  is  restricted 
to  values  less  than  or  equal  to  0^ .  Using”the  same  argument,  we  are 

able  to  show  that 

a  £  a1  -  D(m;0Q;t),  if  a  reject  decision  is  made  on  the  basis  of  Nt=m 

I?  £  £'  =  CfmjO^jt),  If  an  accept  decision  is  made  on  the  basis  of  N^=m 


Thus,  evaluating  D(m;0;t)  using  0  =  0Q  gives  a  "worst  case"  probability 
for  the  producer's  risk,  a.  Similarly,  C(m;©1;t)  is  a  "worst  case" 
probability  for  the  consumer's  risk,  Since  0  remains  unknown,  no 
smaller  bounds  may  be  found  for  a  and  gt. 

d.  The  PFTT  Technique  Using  Alternate  Values  of  MTBF. 

See  page  23  of  this  document. 

e.  The  PST  Technique. 

See  page  26  of  this  document. 
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PROOF*  Let  Dj  =  ^  for  1  =  0,  1. 


Then 


L(x*j »  •X2I  *yl®^  5^ 
L(x1 »  x2*  *  *  * J  xr  J  ®o  f  ^ 

e-^Dl  |l 

Uo  1-1  /  i=1 


-rU„ 


Xj! 


Z  *1 

,-r»i  u/=1 


-rU0 


£■’ 


{See  para  1“  above  r) 


.-4,->(£)n 
=  ^?$" 

e^(^)Al 

f&l  (s)/-! 

e"(^  (fe)D/n'- 

=  P{«t  =  "le=  °l) 

p{Nt  =  n|C  =  0o] 


(Since  n  =  “^i) 


(Since  t  ”  rt#) 


which  completes  the  proof. 
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^  •.  <' r  -A^. .«£a<v;  *y  ^  v-  *•* 


DERIVATION  OF  PROBABILITY  RATIO  SEQUENTIAL  TEST  (IRST) 


1.  Definition  of  Likelihood  Function! 

First,  assume  that 

(The  probability  of  erectly) 
failures  occurring  vben  j 
and  0  are  unknown  s 


P(i;Sjt) 


1  (l)x  Cti 

=  oi«L  =|x 


Nov,  assume  that  successive  obaervetional  values  on  (i  =  1,  2,  . ..,  n) 

are  obtained  for  n  independent  trials  vbere  each  trial  is  e  fired  time 
interval  of  length  tC  (That  is,  r-j  failures  occur  in  tbe  first  time  interval 
of  length  t*,  rp  failures  occur  in  the  second  time  interval  of  length  etc.) 
Then  tbe  likelihood  function,  L,  defined  by 

L(*p  x2,  ...,  x^Q; t)  =  P(r1;Cjt/)*P(x2;0jt#)**'P(xn;0;ti) 


yields  tbe  probability  that  tbe  sample  (x^,  r-*  ...  x^  would  occur  in  erectly 
that  order,  if  tbe  true  MTBF^  0, is  known.  Letting  U  =  g  end  using  tbe 
Poisson  probability  function  (above),  tbe  following  formula  is  derived  for 
tbe  likelihood  function: 


L(x 


V  2* 


.jOjt) 


= 


£ 1 


-u 


r0I 


£1 


ix. 

=  e-^  U  1=1  1 


i=1 


2.  Probability,  Rgt^o  Theorem: 
Consider  the  ratio 

R  =  L  (*1 »  x2*  *■"> 

L  (x-,  ,  x2,  ...»  Xy^jt) 


V 


which  is  the  ratio  of  the  probability  of  obtaining  the  observed  sample 
assuming  0^  is  true  to  tbe  probability  of  obtaining  that  same  sample  assuming 
Q0  is  true.  Now  letting  t  =  rt;,  and  x-j  +  x2  +  ...  +  ij.  =  n,  it  will  be 
proved  that 

R  = 


-  p("t  -  °1  0*  a,} 

p{Kt  =  n|0=  ' 


°o) 
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Slate 


%  • 


p  fyt  g  a  U  g  »i) 

p(Ht»nU=0o\ 


if  R>  1,  then  tbe  probability  of  obtaining  exactly  n  failures  in  time  t  is 
greater  under  tbe  assumption  that  Qu  is  true  than  it  is  under  the  assumption 
that  ©Q  is  true;  hence,  ve  tend  to  believe  that  0  =  ftj.  Similarly,  if 
this  ratio  is  lass  than  unity  ve  tend  to  believe  that  0  =  fl^.  Tims,  constants 
A  and  8  must  be  found  so  that  on#  may 


(i)  Reject  if  R2A>1 
(ii)  Accept  if  R<B<1 


consistent  with  tbe  prescribed  consumer  and  producer  risks  u  and  0  .  Exact 
determination  of  the  constants  A  and  B  baa  not  bean  accomplished;  however, 
it  has  bean  proven1  that  these  risks  will  not  be  violated  if  we  use 


B  * 


with  o i  and  /B  defined  by  the  expressions 

*  P  (n^  £  1  0  >  =  producer  risk 

fi  =r{Nt<  njloi  o,}  =  consumer  risk 

where  n1  corresponds  to  the  value  of  n  under  condition  (i)  above  and  n?  is 
tbe  value  of  n  vben  condition  (ii)  takes  place.  (NOTE:  In  these  express  ion^, 
©o  and  ©1  are  assumed  fixed  and  known,  whereas  0  Is  assumed  fixed  but  unknown. ) 

Thus,  the  following  accept /reject  criteria  may  be  established : 

(«)  R»J.ct  if  B  >  - 

(b)  Aocpt  if  R  <  — — 


See  Wald ’a  "Sequential  Analysis"  published  by  John  Wiley 
&  Sons,  Inc.,  Chapman  and  Hall  Ltd.,  London  (1947) 
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tnd  taka  an  additional  obecrvation  if 


<  R< 


i  -  A 

oc 


The  particular  deciaion  points  nay  be  fourd  in  advance  by  expressing  taat 
time  aa  a  function  of  m  ^  ,  q0,  G, ,  and  n.  This  is  accomplished  as 
follows.  Under  condition  (a)  above 


R  * 


p{N«  =  n  I  0  =  0,} 

p(b;  =  d|o  =  oo> 


> 


oC 


But  la t ting  d  =  Qo/Ol 

p{Ht  =  nl  0=  ft,) 

P{«t  =  n|  0= 


Dl 

7^W7  1 


= 


Thus, 


dn  a1 '  e, 


O'*) 


O  ' 


Hence, 

n  In  d  + 


and  finally 


"in  l1  ~^  )  -  "(in  d)l 


t  (a,):  <  0, 

Similarly,  under  condition  (b)  above 

<a2>*.  >  °o  p”(r^r)  ~ n(ln  d)] 


that  is,  (Ai)  gives  maximum  test  time  for  rajectance  if  n  or  more  failures 
have  occurred  in  time  tn,  and  (A2)  gives  minimum  test  times  for  acceptance 
if  n  or  less  failures  have  oc cured  in  time  t^ 
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TRUNCATION  OF  POISSON  SEQUENTIAL  TEST  (PST) 


It  was  noted  in  Chapter  3  of  this  document  that  the  PST  pro¬ 
cedure  (unlike  the  PRST)  will  not  continue  indefinitely,  Event¬ 
ually  a  point  shall  be  reached  where  the  number  of  failures 
permitted  for  acceptance  ia  one  less  than  the  number  of  failures 
specified  for  rejection  for  the  tame  amount  of  operating  time, 
at  which  point  tha  test  will  necessarily  end.  However,  the  tast 
may  be  truncated  much  earlier  as  a  result  of  the  following  con¬ 
siderations! 

The  specific  decision  criteria  which  is  derived  from  the  use 
of  the  F3T  procedure  takes  the  following  form* 
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Col  1  _ Col  2 _ _ Col  3 _ _ Col  4 


Test 

Time 

Accept  If  Quantity 
of  Failures  (X) 

Eaual  or  Less  Than: 

Reject  If  Quantity 
of  Failures  (X) 

Eou&l  or  Greater  Than 

Continue  if  Quantity 
of  Failures  (X) 
Satisfies 

Ti 

xi 

Xx  <  X  < 

T2 

X2 

xx  <  x  <  x2 

TJ 

*3 

X,  <  X  <  X4 

T4 

 X4 

X,  <  X  <  X4 

T5 

X5 

Xs  <  X  <  X6 

X6 

<  x  <  xfe 

etc. 

etc. 

etc. 

etc. 

NOTE:  Although  it  is  not  necessarily  ^ne  that  acceptance/rejection 
points  will  alternate  with  successive  T  1 s,  as  they  do  In  the 
above  table,  whether  or  not  this  alternation  occurs  has  no 
bearing  on  what  follows.  The  alternation  merely  makes  the 
truncation  procedure  easier  to  follow. 


A  condition  may  now  be  specified  for  terminating  this  test  at  (or 
any  other  for  k  £  3;  or,  more  generally,  at  any  reject  point  past  the 
first  one).  First,  it  is  obvious  that  a  condition  for  termination  would 
not  be  necessary  unless  for  each  k  (k  =  1,  2,  ...  5)>  at  test  time  T  , 
the  conditions  of  Col  4  were  true.  If  denotes  the  number  of  failures 
in  time  T,  and  if  we  are  in  the  continue  test  region  at  time  T^,  then 
the  following  events  must  have  taken  place: 

h  <  %  <  h] 
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i2  =(xx  <  Nt^  ♦ 

V  VT2<X4* 

A4  ={X3  <  NT  +  NT  -  T  <  *4} 
J  4  5 

a5  =ih  <  nt4  +  “Tj  -  t4  <  x6  } 


Now  can  only  take  on  a  finite  number  of  Integral  values  between  Xi 


and  I^.  Let  them  be  a^,  a^, 


aQ^*  Choose  a^  (first)  and  write 


Ai 

Once  is  specified,  -  t  )  0D^  on  a  integral 

satisfy  the  condition^  <  aL  +  N?  T  <  X^.  Let 


values  in  order  to 


them  be  a. 


IV  i2*  alj^ 

xn  ={lir2  -  tx  =  an^ 


.  Choose  a^  (first)  and  write  the  event 


Continue  in  this  manner,  that  is,  assume  a^^  failures  occurred'  in  time  T 


and  au  failures  occurred  in  time  T2  -  T^  then  am,  a112,  ...,  * 
the  only  possibilities  for  the  qumbers  of  failures  occurring  In  time 


*3 


are 


in  order  to  satisfy  the  inequality^  <  +  «u  *  Ny  T  <  X4|  so  that 

one  may  first  choose  the  event 


Xlll  ■  “t3  -  T2  ■  alll 

Eventually  we  derive  oge  particular  way  that  an  equipment  can  stay  in  the 
continue  test  region  up  to  time  T^,  namely,  by  each  of  the  following  events 
occurring  In  successive  time  intervals: 
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A.  =/n^  =  a,) 

L  \  li  L> 

Au  ={n(t.  -  t  )  =  alli 
Alll  ={n(t3  -  T2)  =  aul] 


AIV  ={N(T4  -  T3)  -  aIV  } 

**  =(v5  -  t4)  = 


Now,  it  is  well  known  that  for  the  exponential  distribution  successive 
time  intervals  are  independent,  that  is,  the  probability  of  failure  in 
one  interval  is  not  affectad  by  the  number  of  failures  that  occurred  in 
a  previous  time  interval.  Thus,  we  may  writa 
Pi  =  and  ^  x  and  and  A^y  and  Ay} 

=  PM  '  p(AuV  p(AinV  p(AivV  PM 

=  pW 

If  A^  is  changed  to|A  1  0  =  0^}  then  P  will  give  the  probability  that 
equipment  as  bad  as  0^  would  have  stayed  in  the  continue  test  region  tnrougn 
time  in  exactly  the  manner  specified  by  the  A  's.  Then  by  summing  all 
possible  P^1  s  that  can  be  derived  in  tnis  way,  one  can  compute  tne  prob¬ 
ability  that  equipment  as  bad  as  0^  would  have  remained  in  the  continue 
test  region.  Thus,  one  may  terminate  the  test  at  time  where  this  final 
probability  Is  less  than  or  equal  to  (the  predetermined  risk  that  un¬ 
satisfactory  equipment  will  be  accepted). 

Because  of  the  cumbersome  notation  required  Ao  give  a  theoretical 
explanation  of  this  truncation  procedure,  only  one  of  the  many  possible 
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ways  of  reaching  (without  causing  an  accept/reject  declsiorOwae  given. 
Another,  way,  of  course,  is  to  chooee  a,  at  the  firet  step,  then  choose 
at  the  second  etep  (instead  of  a^).  Ultimately,  this  leade  toi 

Ai  ={NTl  • 

A12  ={N(T2  -  Tj)  =  *12! 

A121  ={N(T3  -  T2)  =  a12ll 

AI211  ={N(T4  -  =  a121l) 

A 12111  ={*(T5  -  T4)  =  a1211l) 

In  fact,  one  may  derive  all  the  poseible  waye  that  thie  may  happen,  where 
all  avants  are  expreseed  in  terms  of  the  successive  tima  Intervals  between 
decision  points.  A  concrete  example  will  serve  to  convince  those  in  doubt: 


Test 

Time 

Accept  if  Quantity 
of  Failures  less 

Than  or  Zqual  to: 

Reject  if  Quantity 
of  Failures  Greater 
Than  or  Equal  to: 

Continue  If  Quantity 
of  Failures  Equal  to: 

Tl=160 

0 

— 

1  or  2 

V195 

-- 

3 

1  or  2 

- - - 

—  - - 4 - 

~  ^7-01  0  - 

T  =300 

4 

1 

— 

2,  3,  or  4 

V385 

— 

5 

2,  3,  or  4 

V430 

2 

— 

3,  4,  or  5 

T?=485 

— 

6 

3,  4,  or  5 

_ £iS*_ 

_ SISj - 

etq, 

etc. 
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The  above  table  was  derived  by  using  the  PST  decision  rule  using  20$ 
risk  conditions.  Let  us  attempt  to  truncate  the  test  (by  the  procedure 
explained  above)  at  test  time  -  285  hours,  assuming  that  0^  =  100  hours. 
The  main  problem  is  to  list  all  possible  ways  of  staying  In  the  continue 
test  region  up  to  and  including  time  T^.  The  following  schematic  diagram 
shows  how  simple  this  is: 


QUANTITY  OF  FAILURES  THAT  MAY  OCCUR 

In  Time 

Tx  -  0  =  160 

1 

or 

2 

In  Time 

VT1  =?  35 

If  1  at  time 

Q or 

T^  then 

1 

If  2  at  time  T^  then 

0 

In  Time 

T3-T2  =  90 

ft  i+0(  =  1) 

at  time  T? 
then 

0,  1,  or  2 

If  1+1 (=2)  at 

time  T2  then 

0  or  1 

If  2  +  0(=2)  at  time  T^  then 

0  or  1 

Thus,  there  are  7  possible  ways  of  remaining  in  the  continue  test  region 
through  test  time  T^: 


QUA 

NTITY  OF  F 

AILUB 

ES  THAT  MAY  OCCUR 

In  Time 

T  -  0  =  160 

1 

1 

1 

1 

1 

2 

2 

In  Time 

T2  -  Tj  =  35 

0 

0 

0 

1 

i 

0 

0 

In  Time 

T3  -  t2  =  90 

0 

i 

2 

0 

1 

0 

i 

The  question  now  to  be  answered  is:  What  le  the  probability  that  equip¬ 
ment  as  bad  as  0^  =  100  hours  MTBP  would  behave  in  any  of  these  seven  ways? 
Tnis  is  answered  by  computing  each  of  the  seveq  column  probabilities,  and 
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then  eumming  the  seven  results.  Giving  the  details  for  Column  1  only  we 
get: 

Col  1:  p{ni6q  =  1  and  r  o  and  =  o} 

=  P{N160=1V  P(n35  =  °)‘  P(n90=  °} 


=  (.32)  (.71)  (.41) 


=  .093 
Similarly, 


Col  2: 

.083 

Col  3: 

.038 

Coi  4: 

.032 

Col  5: 

.030 

Coi  6: 

.076 

Col  7: 

.068 

Tne  sum  of  these  probabilities  is  .42  =  42$.  Since'  the  final  result 
exceeds  the  original  risk  condition  (20$)  it  is  concluded  that  it  is  not 
possible  to  truncate  the  test  without  contradicting  the  prescribed  risk. 
(It  can  be  shown,  however,  that  truncating  at  3.85  gives  a  result  which 
lb  compatible  with  20$  risks.) 
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DEFINITION 

1.  HE  LIABILITY .  The  probability  of  failure-free  operation  in  a  given 
amount  of  operating  time.  (NOTE:  This  is  a  '’mathematical'1 ,  not  an 
‘’engineering"  definition.  The  "engineering1’  dafinition  would  add  such 
phrases  as  "when  operated  under  specified  conditions",  or  "when  used  in 
the  intended  environment".) 

2.  MTBF  (Mean-Time-Between-Failure) .  The  arithmetic  average  of  all 
failure-free  operating  intervals  during  the  "useful  life"  of  the  equip¬ 
ment.  As  used  herein,  MTBF  has  meaning  only  if  the  failure  behavior 
can  be  described  by  the  exponential  or  Poisson  distributions. 

3.  USEFUL  LIFE.  Tne  period  (between  the  end  of  the  burn-in  stage  and 
tne  beginning  of  the  wear-out  stage)  during  which  a  constant  failure 
rate  is  exhibited. 

4.  FaILUPE  RATE.  The  reciprocal  of  MTBF. 

3.  CONSUMER'S  RISK.  The  probability  that  unsatisfactory  equipment  will 
be  accepted. 

6.  PRJDL CZR 1 S  RISK.  The  probability  that  satisfactory  equipment  will  be 
rejected. 

7.  MTBF  UPPER  BOUND.  A  value  of  MTBF  wnicn  represents  the  required  value 
or  objective.  Values  of  MTBF  greater  than  or  equal  to  tne  MTBF  upper 
bound  are  considered  satisfactory. 

8.  MTBF  LOWER  BOUND.  A  value  of  MTBF  wnicn  represents  an  unacceptable 
value.  Values  of  MTBF  less  than  or  equal  to  the  MTBF  lower  bound  are 
considered  unsatisfactory. 

9.  ZONE  OF  INDIFFERENCE.  Values  of  MTBF  (greater  than  the  MTBF  lower 
bound  and  less  tnan  the  MTBF  upper  bound)  which  are  considered  neitrer 
satisfactory  nor  unsatisfactory.  The  probability  of  accepting  (reject- 

such  equipments,  is  permitted  to  r  i se^ higher.  thaj^ ^elp^djgter^nec^^  ^ 
“  consumer  (producer!  risk  level. 

10.  DISCRIMINATION  RATIO.  The  ratio  of  "MTBF  upper  bound"  to  "MIBF 
lower  bound". 
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SYNTAX 


a  :  Producer’s  risk, 
p  :  Consumer*  s  risk. 

6  :  True  MTBF 

0^:  MTBF  Upper  Bound 

0^:  MTBF  Lower  Bound 

0*:  MTBF  Requirement  (when  upper  and  lower  bound  values  have  not  been 
cnosen) . 

d  :  Discrimination  Ratio  (0  /0,). 

o  i 

N^:  The  number  of  failures  that  occur  in  time  t. 

P(x;0; t) :  The  probability  that  N  is  equal  to  x,  if  0  is  known. 

C(x;0;t):  The  probability  that  Nfc  is  less  than  or  equal  to  x,  if  0  is 

known. 

D(x;0;t):  The  probability  that  N^  is  greater  than  or  equal  to  x,  if  0 
is  known. 

PFTT:  Poisson  Fixed  Time  Test. 

PST:  Poisson  Sequential  Test. 

PRST:  Probability  Ratio  Sequential  Test. 

B  is  a  statement  such  as  ”0  £  0*"  ,  where  0*  is  a  fixed,  known 
value  and  0  is  a  fixed  unknown  value. 
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SECTION  VI 

VERIFICATION  OF  QUANTITATIVE  RELIABILITY 
REQUIREMENTS  -  DECISION  CRITERIA 


FORE  VO  RD 


This  section  presents  guidance  to  Electronic  Systems  Division  (ESD)  SPOs 
in  the  statistical  aspects  of  planning  equipment  reliability  demonstrations. 

For  one-of-kind  equipments  with  "high"  mean-time-between-fai lures  (MTBF) 
requirements,  the  present  Table  I  of  MIL-R-26474  leads  to  lengthy  test  times 
before  a  decision  can  be  made  on  equipment  reliability.  This  section  offer 
methodology  for  devising  alternate  approaches  to  reliability  demonstratio  . 
In  addition,  a  quantification  of  the  risks  involved  in  equipment  reliability 
decision-making  is  presented. 

As  additional  thinking  is  developed  on  equipment  reliability  demonstra¬ 
tion,  it  is  planned  to  revise  this  document. 
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Chapter  1 


INTRODUCTION 


This  section  gives  a  straightforward  explanation  of  the  mathematic  fill  tools 
tnat  are  avaiaADxe  fur  demun»trat,ing  the  lex^-ability  of  electronic  system* /equip¬ 
ment  through  testing  Most  presentations  of  these  techniques  fail  to  set  forth 
the  underlying  ideas  and  neglect  to  exemplify  their  logical  coherence,  It  iB  for 
this  reason  that  confusion  prevails  as  to  the  meaning  and  value  cf  a  statistical 
assessment  of  Mean  Time  Between  Failure  (MTBF)  and  other  Reliability  indices 
The  purpose  of  this  sectfon  is  to  uhow  that  these  tools  are  not  mere  compilations 
of  arbitrary  rules  to  be  learned  by  rote ;  nor  are  they  techniques  to  be  acquired 
by  imitation  and  used  without  consideration  of  the  equipment s  involved;  but 
rather  they  must  be  developed  logically  from  a  few  fundamental  principles  in 
accordance  with  the 

(1)  Number  of  models  available  for  testing; 

(2)  Contractual  MTBF  requirement;  and 

(3)  Determination  of  risk  levels  that  are  consistent  with  schedules, 
cost  of  testing,  test  environment,  and  equipment  characteristics 

Relatively  few  tools  are  needed  to  accomplish  this  task,  and  they  are  amazingly 
single  to  apply  in  the  special  case  of  electronic  equipments.  It  is  only  when 
these  techniques  are  stated  generally  (so  that  they  apply  to  quality  control, 
biological  studies,  insurance  considerations,,  etc.)  that  they  take  a  complicated 
form. 

These  introductory  remarks  should  make  it  clear  that  this  section  stresses 
the  fact  that  no  military  specification  can  provide  optimum  criteria  upon  which 
to  assess  t be  reliability  of  electronic  systems.  Tet,  many  programs  hare  been 
using  MH-R-26V74  in  exactly  this  way  Table  I  of  MXL-R-26474  has  been  cited 
in  many  equipment  specifications  despite  its  unreasonableness  when  the  number  of 
models  available  for  test  is  small  and  the  MTBF  requirement  is  high •  For  example, 
if  the  MTBF  requirement  is  1,000  hours.  Table  I  requires  3,000-10,000  hours  of 
testing,  and  only  when  the  equipment  is  extremely  better  or  extremely  worse  than 
the  contractual  requirement  will  an  accept/reject  decision  be  made  at  the  3,000 
hour  point.  If  the  true  MTBF  lies  in  the  5^0-1,000  hour  range,  it  is  quite  likely 
that  6,000  hours  of  testing  or  more  will  be  required.  If  only  two  models  were 
available  for  testing,  this  period  of  indecision  could  lest  aix  months  or  more. 

In  fact,  it  is  not  unusual  to  see  equipment  specifications  cite  Table  I  in  one 
paragraph,  thereby  requiring  (poeaibly)  three  to  six  months  of  indecision,  but 
then,  in  another  paragraph ,  provide  only  75  days  (or  less)  of  acceptance  testing 
during  which  time  reliability  testing  is  to  be  conducted.  The  result,  of  course, 
is  that  Table  I  is  unenforceable  and  there  is  essentially  no  reliability  demon¬ 
stration  criteria  in  for^e.  Under  these  conditions,  reliability  testing  is  little 
more  than  a  debugging  exercise,  and  we  are  forced  to  accept  whatever  MTBF  is 
achieved 
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Another  indication  that  there  la  a  lock  of  unoeratamilng  o.  e  .  *s 
equipment  specification,  frequently  state  "The  contractor  shall  prove  to  a  90* 
confidence  thlt  the  equipoent  KTBF  equals  or  exceeds  (s»y)  700  hours  In  occord- 
vlth  Table  I  of  MTL-R-2647U" .  It  Is  quite  unfortunate,  but  true,  that  fev  are 
avare  of  the  fact  that  Table  I  Is  based  on  giving  confidence  that  the  true  MTBF 
is  not,  lover  than  one-half  the  required  700  hours.  It  is  even  Inadvisable  to 
state  "90*  confidence  that  the  equipment  KTBF  exceeds  350  hours  ,  sctwe  this 
Implies  that  our  90*  confidence  Is  associated  vith  some  figure  greater  than 
35O.  Actually,  the  confidence  figure  reduces  considerably  as  ve  progress 
upwards  free,  350  to  numbers  close  to  700.,  when  Table  I  criteria  Is  stipulated, 
if^pears  that^he  expression  "90*  confident  that  the  true  KTBF  Is  not  lover 
than  350  hours"  Is  the  least  misleading,  but  evej  this  formulation  Is  an 
interpretation  of  another  mathematical  statement1.  When  confuted  with  these 
££5* In  immediate  reaction  Is  to  want  90 *  confidence  that  700  hours  has  besn 
achieved,  that  Is,  Table  I  appears  unsatisfactory.  Such  a  change,  however,  may 
nullify  the  original  purpose  of  the  demonstration  because  of  the  length  of  test 
time  required  before  decisions  can  be  made  The  higher  these  figure”  are  se  , 
the  longer  it  will  take  to  accept  (reject)  equipment  which  Is  satisfactory 
(unsatisfactory)  ,  hence,  they  must  be  set  In  accordance  with  the  consequences 
of  delayed  decisions 

Besides  providing  educational  material  for  those  concerned  with  or  affected 
bv  reliability  programs,  this  section  should  serve  as  a  working  guide  or 
Bell  ability  Monitors.  Although  demonstration  plans 

particular  system/equipments  under  procurement,  homemade  plans  which  lack 
S£E<  the  risks  Evolved  are  to  he  discouraged  This  pasqfclet  Is  pr^rily 
concerned  with  the  methods  available  for  keeping  risks,  “well  as  test  time,  at 
minimum  levels.  This  is  accomplished  via  the  concept  of  Sequential  Testing 
which  was  first  developed  by  Wald3.  Unfortunately,  neat  formulas  were  developed 
for  devising  sequential  tests,  causing  workers  In  the  field  to  soon  forget  the 
baslcconcepts  which  the  formulas  were  based.  These  basic  ideas  are  fully 

discussed  herein  -  the  formulas  are  only  mentioned  in  order  to  discuss 
limitations.  Table  X  is  Just  one  specific  instance  of  thsse  formulas  *  with  ris 

quoted  term*  are  explained  In  Chapter  3)  Once  this  Is  understood  by  everyone 
concerned  (and  only  elementary  algebra  and  analytic  geometry  are  prerequisite  for 
understanding  these  Ideas)  realistic  demonstration  criteria,  which  is  capable  of 
being  enforced,  may  be  devised  to  fit  particular  programs.  It  is  even  conceivable 

that* demonstration  criteria  may  be  stipulated  and  (more  clterora 

during  Category  I  testing,  with  full  verification  occurring  earl£  In  the  Category 
latest  program.  Usually  formal  reliability  testing  1.  specified  only  a.  a  part 
£  fl^ll^f^ance  £tt^g  when  It  1.  quite  late  to  make  significant  change,  to 


1  Tfe#  1*  MjWiesd  in  Chnpw  3.  “tk#  jrobnbiliiy  of  .ccoptlnf  oqulpo-nt  wtth  MTBF  -  MO 

U  10%  («  war 

*  MIL-R-S6667A  kw  »  pUn  which  o»«i  2 '3  *h»  rnqijomonf  (Method  2). 

A  only*  >«**  by  A.  Wold.  Jchn  Wiley  A  Sono.  Inc  ,  Cbnpmon  ond  Hill  Ltd..  London  (1947). 
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the  reliability  of  ths  equipments.  If  Reliability  truly  is  &  function  of  the 
design,  verification  of  this  design  feature  mast  be  made  as  early  as  possible  in 
the  program. 

In  order  to  give  a  better  under standing  of  this  presentation,  included  in 
Chapter  2  is  the  basis  for  using  probabilistic  theories  (namely,  the  cumulative 
Poisson  distribution)  for  predicting  the  reliability  of  electronic  equipments 
Also  included  in  Chapter  2  is  the  establishment  of  measures  of  reliability  such 
as  KEEP  and  "failure  rate" .  Those  familiar  with  these  ideas  me/  start  with 
Chapter  3  after  a  brief  scanning  of  Chapter  2. 

Some  final  words  of  caution  before  entering  upon  the  main  subject  matters 
Any  statistical  assessment  of  Reliability  is  meaningless  unless  ground  rules  have 
been  clearly  established  for  counting  "failures"  and  measuring  "operating  time"  . 

In  order  to  quantify  Reliability,  it  was  necessary  to  leave  certain  concepts  un¬ 
defined  (namely  those  Just  quoted),  to  be  defined  for  the  particular  equipments 
under  consideration.  A  common  mistake  is  to  assume  that  these  concepts  have  been 
previously  defined,  and  that  it  is  sufficient  to  simply  stats  the  numerical 
reliability  requirements .  Moreover,  definition  of  these  concepts  cannot  be  accom¬ 
plished  hy  placing  such  adjectives  as  Inherent,  Achieved,  Delivered,  or  Operational 
before  the  word  Reliability.  These  modifiers  are  useful  for  conversational  pur¬ 
poses  but  they  cannot  take  the  place  of  unamb iguoualy  specifying  relevant  and 
non-relevaat  failure  classifications,  and  precisely  defining  what  is  meant  by 
operating  time.  Ths  methods  employed  here  assume  that  these  ground  rules  have 
been  precisely  defined.  It  is  also  mandatory  that  a  failure  reporting  system  has 
been  established  to  record  and  analyze  failures  that  occur.  This  section  is 
concerned  vith  measuring  reliability  during  the  constant  failure  rate  zone  only; 
and  although  it  is  the  contractor’s  responsibility  to  thoroughly  debug  the  equip¬ 
ment  before  KTBF  demonstration  begins,  ve  cannot  close  our  eyes  to  the  types  of 
failures  that  occur . 
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CHAPTER  2 


ESTABLISHING  A  SCIENTIFIC  DEFINITION  OF  RELIABILITY 
(THE  CUMULATIVE  POISSON  DISTRIBUTION  FUNCTION) 


2.1  Sxaxemenc  oi  the  Problem.  The  fxiot  attempc  to  qu&nxify  reliability  vae 
"the  probe bility  that  e  device  will  perform  its  intended  function  for  the 
period  of  time  required  under  the  opereting  conditions  encounx*  red".  A  few 
moments  of  quiet  thought  should  be  enough  to  convince  anyone  th  t  this 
definition  is  hardly  scientific.  The  expressions  "perform  its  imended 
function"  end  "opereting  conditions  encountered"  prevent  us  from  thinking 
clearly  when  attempting  to  essign  numbers  from  0  to  1,  es  the  mathematical 
theory  of  probability  requires.  Learning  e  lesson  from  Euclid,  however  (who, 
without  ca-ing  where  or  how  his  points  and  lines  would  be  used,  nevertheless 
built  his  system  of  geometry)  we  might  simply  drop  the  expression  "opereting 
corditions  encountered"  and  proceed  undaunted.  It  might  el3o  make  the  theory 
more  interesting  to  those  who  leter  epply  it,  since  they  would  heve  the 
important  task  of  simuieting  the  expected  opereting  conditions.  But  we  still 
heve  troubles  -  the  expression  "perform  its  intended  function"  is  vague, 
meaning  different  things  to  different  people.  We  could  convert  this  expres¬ 
sion  to  "not  fail"  and  such  e  transformation  might  et  first  eppear  to  remove 
ambiguities t  but  further  examined on  reveels  thet  the  gein  is  superficial,  and 
so  (learning  enother  lesson  from  pest  scientific  progress)  we  conclude  that  we 
are  struggling  with  an  indefineble  concept.  We  heve  enother  one  of  these 
elusive  creetures  in  our  definition,  namely  "time",  and  although  we  already 
know  how  to  meesure  time,  we  simply  cannot  consider  all  calendar  time  (for 
example,  repair  time).  Here  egain,  we  look  et  Euclid  and  see  that  he  worried 
little  ebout  the  definition  of  e  point.  Rather,  he  left  it  undefined,  so 
thet  his  theory  could  epply  ju3t  ea  well  to  e  molecule,  e  pencil  point,  e 
spot  of  gresa,  or  even  Pike's  Peak.  Possibly  our  theory  will  epply  regardless 
of  how  the  users  of  the  theory  define  "failures"  and  "operating  time".  Our 
definition  now  reads-  "The  probability  thet  a  device  will  not  fell  for  the 
opereting  time  required",  and  we  are  now  interested  in  establishing  measures 
of  reliebility  in  terms  of  "operating  time"  and  the  number  of  "feilures"  that 
occur,  where  the  quoted  terms  are  left  undefined. 

2  2  Establishing  Measures  of  Reliability.  Let  us  essume  thet  it  hes  been 
(unambiguously)  specified  how  to  determine  that  e  failure  has  occurred,  and 
when  the  equipment  is  in  an  opereting  state.  We  could  then  operste  the 
device  until  e  feilure  occurs,  correct  the  feilure,  and  continue  operetion 
until  the  next  feilure „  again  take  corrective  ection,  and  so  on.  After  e 
certain  period  of  (opereting)'  time,  we  could  look  back  and  count  the  number  of 
failures  that  occurred,  say  x.  Can  x  be  used  es  the  measure  of  reliebility? 
No,  since  x  could  be  larger  or  smaller  depending  on  the  amount  of  opereting 
time  t.  How  ebcut  using  the  ratio  t/x  es  our  measure,  thet  is,  the  total 
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opersiinr  time  divided  by  the  total  number  of  failures?  Intuitively,  we  would 
consider  this  number  ea  the  "mean  time  between  failures "  (MTBF)  that  uss 
demons treted  during  this  operating  period.  (The  reciprocel  of  this  number, 
which  we  denote  by  X  *  might  reesonebly  be  celled  the  "failure  rete",  that  ia, 
X  =  x/t  =  '/VtTBF.)  But  whet  does  this  tiey  ebout  the  probability  of  not 
feiling  during  future  periods  of  operation?  Can  we  use  this  figure  to  predict 
the  number  of  failures  that  will  occur  in  future  opereting  periods?  For 
example,  if  a  device  hes  five  feilurea  in  1,000  houra  of  operation,  we  derive 
the  figure  200  hours  KTBF.  Does  this  mean  thet  we  can  expect  e  failure  to 
occur  approximately  every  200  houra  in  the  future  7  One  readily  aeea  problems 
here,  and  fortunately  this  problem  has  been  studied  for  centuries  by 
probability  theorists  ao  thet  a  logical  approach  may  be  given.  The  following 
paragraphs  give  the  basic  assumptions  contained  in  this  approach. 

2.3  The  Basis  for  the  Definition.  Besides  games  of  chance,  certain  physical 
phenomena  (such  es  the  occurrences  of  defectives  produced  in  a  manufacturing 
process,  errors  in  clerical  operations,  and  most  important,  occurrences  of 
failures  in  electronic  devices  during  s  "certain"  period  of  their  life) 
exhibit  e  so  called  statistical  regularity  which  enables  us,  in  spite  of  their 
unpredictable  nature,  to  predict  their  behavior  "in  the  long  run ".2  a  good 
example  of  e  mass  phenomenon  suitable  for  the  application  of  this  theory  ia  the 
inheritance  of  certain  characteristics;  for  example,  the  color  of  flowera 
resulting  from  the  fertilization  of  large  numbers  of  plants  of  a  given  apeciea 
by  the  pollen  of  e  given  plant  of  the  same  species.  A  further  example  ia  the 
whole  class  of  insured  men  and  women  whose  ages  et  death  have  been  registered 
by  an  insurance  firm.  Still  another  ia  the  class  of  resistors  with  a  certain 
aerial  number  which  is  manufactured  by  e  particular  firm,  whose  length  of  life 
hes  been  recorded.  The  properties  inherent  in  all  of  these  examples  hove  been 
assigned  technical  terms,  such  es  randomness  and  stochastic  independence .  but 
we  have  no  need  of  these  terms  and  shall  not  use  them.  Wa  simply  state  here 
rather  crudely  that,  after  e  certain  "burning-in"  period,  there  is  a 
statistical  regularity  to  the  occurrence  or  non-occurrenca  of  failures  in 
electronic  devices,  and  that  this  regularity  persists  until  the  device  reaches 
old  ege,  that  ia,  the  wear-out  stage.  Vhet  makes  the  situation  even  nicer  is 
that  the  burning-in  stage  is  usually  fairly  short  relative  to  this  period  of 
uniform  behavior.  We  mey  even  give  an  idealized  graphical  picture  of 
occurrences  of  failures  (known  ss  the  "bathtub"  curve)  es  shown  in  Figure  I. 
This  curve  illustrates  thet  the  failure  rate  is  quite  high  during  the  infant 
stage,  then  remains  constant  for  a  long  period  of  time  until  reaching  the 
wear-out  stage.  The  existence  of  this  constant  zona  ia  what  is  meant  by 
statistical  regularity,  and  Justifies  our  uss  of  the  basic  lavs  of  probability 
theory.  A  brief  listing  of  thasa  lavs  (which  shall  be  used  sparingly  in  sub¬ 
sequent  paragraphs)  is  included  in  Attachment  1. 


2  Br la  fly,  this  meene  that  wa  could,  if  wa  wara  ao  lnclinad,  continue  our  aiperimeat  to  Indefinite  lanftha. 
For  example,  in  toaaics  a  carteln  coin,  wa  might  aaa  "heeda”  Oceania*  about  half  tba  time,  Thia  doee  aot 
maan  that  for  aona  finite  number  of  toaaaa  “heeda”  appear  erectly  one  half  tba  tine,  bat  that,  aa  tbo 
number  of  toaaaa  increaaaa  (indefinitely  tba  relative  frequency  of  occur  ranee  of  beada  gate  cloaar  to  1/7. 
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failure  rate 


THE  "BATHTUB"  CURVE 


Figure  1 


^  4  T  he  iJefinitlon-  Probability  theorists  heve  derived  For  us  what  is 
called  the  binomiel  frequency  function  which  applies  to  experiments  which,  in 
addition  to  satisfying  basic  lews  of  probability,  ere  only  interested  in  the 
occurrence  or  non -occurrence  of  an  event  (a  feilure,  for  inLtence).  If  we 
know  the  probability  p  of  the  event  occurring  in  any  one  trial  of  the 
experiment,  so  that  the  probability  of  the  event  not  occurring  is  (1-p),  then 
the  probability  of  obtaining  exactly  x  occurrences  of  the  event  in  n  trials 
defines  whet  is  known  es  the  binomiel  (or  sometimes  the  Bernoulli)  frequency 
function.  (Also  called  the  binomiel  distribution.)  This  function  is  denoted 
by  b(n;x;p)  because  it  depends  on  n,  x,  end  p  where: 

n  =  the  number  of  trials  to  be  made 

x  =  the  totel  number  of  occurrences  of  the  event  in  question 

p  =  the  probability  of  the  event  occurring  in  any  one  trial. 

It  is  mostly  a  mathematical  problem  to  prove  that  this  frequency  function  is: 

b(n;x;p)  =  n!  .  p* *  .  (1-p)n_x 

x! (n-x) ! 

so  that  it  will  not  be  jroven  here.^  To  illustrate  its  use  we  ahall  try  to 
calcrete  the  probability  of  obteining  exactly  7  "two's"  in  100  tosses  of 
a  die,  if  it  is  known  thet  the  prebebility  of  obtaining  a  "two"  in  any  one 
toss  is  ^/6.  We  heve 

b(lOO;7;1/6)  =  *00! _  .  (l/6)7  .  (1  -  l/6)10O~7 

7!  (100-7)!  ' 

=  OC0)(99)(98)(97)(96)(95)(94)(5q3) 

(7)(6)(5)(4)(3)(2)(67)(693) 


=  ? 

ard  obviously  this  is  too  laborious  to  calculate  -  even  after  some  simplifica¬ 
tion.  But  do  not  get  discouraged,  mathematicians  heve  elso  discovered  an 


^  Ihi*  idjv  be  establ.ahed  o  prior’,  oc  by  using  the  observed  relative  frequency  of  occurrence.  For  example,  in 
•oss.ng  ■  die  we  may  be  conecned  wnh  4  "two”  appear. ng  or  net  appearing  on  successive  toeaea.  We  mey 
use  o  1/6  ••  the  a  pricri  probability  or  rrake  1,000  osses  wherein  300  “two’s"  appeared,  thereby  considering 
rhe  [far  .  ion  3/10  as  mo«  -epresenta’i  ve  o'  the  probab.l  ty  p.  In  our  case  we  would  use  the  failure  rate  A 
whub  is  obae~ved  during  tbe  constant  zoae. 

*  Tbe  proof  can  be  found  ;n  any  standard  test  on  probability  theory. 
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approximation  to  the  binomial  frequency  function  which  works  reasonably  well 
when  p  is  small  and  n  is  large 5.  This  is  given  by 

b(n;x;p)  ^  e~nP  (np)x 
x! 

which  is  called  the  Poisson  frequency  function.  (Also  called  the  Poisson 
distribution)  Let's  try  it: 

b(100;7;l/6)  =  e-100/6  (100/6)7 

7! 

=  (-16.67  (16.67)7 
5040 

=  .004  (approximately) 

With  the  use  pf  standard  mathematical  tables  or  e  slide  rule  we  can  make  this 
calculetion  in  e  few  minutes.  But  usually  we  are  more  interested  in  knowing 
the  probability  of  echieving  r  or  less  occurrences  of  e  particular  event  (e 
feilure)  and  this  can  be  found  by  finding  the  probabilities  when  x  =  0,1, 
2,...,r  and  by  adding  all  of  these  results  we  derive  the  probability  of 
obteining  r  or  less  occurrences  of  the  event  in  question.  For  example,  if 
r  denotes  the  number  of  "two's"  which  eppear  on  our  die,  the 

/probability  that  r<  7  in  100  tosses\  =  ^  b(l00;r;1/6)  =  .008 

V  J  r=0 

A  function  so  defined  is  called  a  cumu. etive  distribution  function^.  Thus  far, 
it  might  not  heve  been  hard  to  s$e  the  anelogy  between  these  considerations 
and  our  situetion.  The  event  of  interest  is,  of  course,  a  feilure,  and  the 
obeerved  failure  rete  is  the  probebility  p  (esauming  that  our  obeervetions 
occurred  during  the  constant  feilure  rete  zone  of  the  bathtub  curve).  Our  only 
problem  is  what  is  n?  For  our  purposes,  we  are  interested  in  what  will  happen 


®  In  our  cm*  the  (alive  rete  X  is  usually  a  email  number.  Wa  ahall  apeak  about  later. 

®  Snch  a  function  may  he  defined  for  any  random  variable  r.  (See  Attachment  J.i  Tbe  adjective  "cumulative" 
is  often  inclnded  becanee  certain  frequency  functions,  anch  aa  tbe  binomial  and  Poiaaon  are  aooetlmea  called 
Attribution  functions. 
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over  a  period  of  time  (which  is  a  continuous  variable)  not  what  happens  in 
'V'  independent  observations  (a  discrete  variable).  One  solution  is  to 
tacitly  make  tha  assumption  that  "that  which  holds  in  tha  discrete  case  alao 
holds  in  the  continuous  caae"  ard,  unhes i tantly ,  replace  n  by  t  in  our  formula. 
For  intuitive  justification  we  might  reason  as  follows:  the  n  independent 
trials  could  be  n  intervals  of  ona  minuta  aach;  that  is,  failures  which  occur 
during  any  minute  would  be  considered  to  have  occurred  at  the  arvd  of  that 
minute.  Also,  the  failure  rate  could  be  expressed  in  minutes.  This  has 
tha  effect  of  changing  the  continuous  variable  to  a  discrete  ona,  and  there 
seems  to  be  little  difference  (except,  possibly,  a  philosophical  one).  We 
could  then  proceed  to  empirically  justify  our  actions  by  comparing  theoretical 
results  to  actual  results.  This  having  already  been  done  for  us,  we  may,  with 
electronic  equipments,  replace  np  by  t>.  ,  that  is,  replace  the  "number  of 
trials"  by  "time"  and  the  "probability  of  a  failure  occurring  in  any  one  trial" 
by  the  "failure  rata".  This  change  gives  us  the  cumulative  Poisson  distribu¬ 
tion  function  which  has  become  famous  in  reliability  analysis: 


("probability  that  r  or  fewer\ 
*(  failurea  occur  in  time  t  J 


=  £  e~Xt  (At)X 
x=0  xl 


Since  A 


1/MTBF,  if  we  let  MTBF  =  6,  we  have  the  equivalent  form: 


(probability  that  r  or  fewerl 
^failures  occur  in  time  t  j 


21  e~,/e  (t/Q)x 

x=0  x! 


which  is  the  form  used  for  moat  reliability  calculations.  If  one  ia  interested 
in  the  probability  of  r  =  0  failuraa  occurring  during  the  period  of  time  t,.  this 
expression  reduces  to: 

^Probability  that  0  failures  occur  in  time  t}  =  a*  Xt  =  e-t/8 

But  this  probability  is  exactly  what  our  (verbal)  reliability  definition 
reduced  to  pravioualy,  that  is,  "the  probability  that  a  devica  will  not  fail 
for  the  operating  time  required."  Denoting  this  expression  by  R(t),  we  hava 
thua  accompliahed  our  task,  namely: 


R(t)  =  a*1/0 


A  graph  of  this  function  appears  in  Figure  2. 
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PROBABILITY  OF  SUCCESS 
(ZERO  FAILURES) 


THE  RELIABILITY  FUNCTION 


Figure  2 


11 


Chapter  3 


APPLYING  THE  CUMULATIVE  POISSON  DISTRIBUTION 
TO  MTBF  DEMONSTRATION 


3.I  The  Basic  Tools  We  have  thus  far  developed  formulas  which  have  become 
widely  used  In  reliability  analysis  These  are  the  reliability  function: 


R(t)  :  e  *t/® 


(The  probability  of  zero  failures  during  the! 
operating  time  t,  when  0  =  MTBF  is  known.  J 


Given  t  and  0,  R(t)  gives  the  probability  of  failure  free  operation  during  that 
time  This  function  is  a  special  case  of  the  more  general  cumulative  Poisson 
distribution  function  which  gives  the  probability  of  having  x  or  fewer  failures 
in  a  specified  time,  if  0  is  known  For  simpxiclty  we  denote  this  probability 
by  C(x)Jt  and  letting  t/0  z  U,  it  takes  the  form 


C(x)  = 


e-U  Ur 


r  ■  0 


The  probability  of  having 
x  or  fewer  failures  when  7 
U  -  t/Q  is  known.  J 


Another  formula  which  can  be  derived  from  this  one  is 


D(x)  :  ^1  • 


C(x-l),  if  x>  : 
if  x  --  0 


f  The 

')  ‘ 


The  probability  of  having] 
or  mere  failures  when  V 
t/0  is  known.  J 


All  were  derived  from  the  original  Poisson  distribution  function 


P(x)  = 


.-U 


The  probability  of  having] 
ctly  x  failures  when  V 
t/5  is  known  J 


exa> 

U 


A  table  of  values  (rounded  off  at  3  decimal  places)  for  P(x),  C(x),  and  D(x),  for 
various  values  of  U  and  x,  is  given  in  Attachment  4  All  of  these  formulas  were 
derived  from  the  basic  assumption  that  after  a  short  debugging  period,  the  failure 
rate  of  electronic  equipments  remains  constant  until  reaching  the  wear-out  stage . 
Now  we  wish  to  show  how  these  formulas  can  be  used  for  the  prediction  of  MTBF 
through  demonstration  testing. 


3.2  The  Technique  -  The  Meaning  of  Confidence  We  mentioned  in  Chapter  2  an 
example  of  a  device  which  (after  debugging)  exhibited  5  failures  in  1,000  hours 
of  operation,  and  we  asked  if  this  demonstrated  MTBF  of  1,000/5  =  200  hours  is 
meaningful  for  prediction  purposes.  When  a  problem  defies  solution,  one  useful 
technique  is  to  rephrase  it  In  fact,  henceforth,  we  must  develop  the  ability 
to  rephrase  all  such  questions  in  terms  of  the  language  of  our  formulas.  Let's 
try  this.  We  might  change  our  question  to.  What  is  the  probability  of  a  device 
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having  5  or  less  failures  in  1,000  hours  of  operation,  if  its  true  MTBF  is  less 
than  200  hours?  This  is  better,  but  still  not  satisfactory  since  our  formulas  re¬ 
quire  that  we  know  how  much  less.  So  let's  pick  a  specific  number  less  than  200, 
say  100  hours.  (This  number  might  be  called  the  "absolute  minimum  MTBF".)^  Now 
we're  in  business  -  we  simply  calculate  U  -  1,000/100  :  (test  time)/(abs.  min. 
MTBF)  -  10,  then  find  U  :  10  in  the  tables  (Attachment  U)  and  then  reading  across 
from  x  ‘  5  ve  see  that  C(x)  “  067  =  7$  approximately  That  is,  a  device  whose 

true  MTBF  is  100  hours  has  only  about  a  7$  chance  of  having  5  or  less  failures  in 
1,000  hours  of  operation.  We  might  then  say  that  under  this  demonstration  we  are 
100$-7$  r  93$  confident  that  the  true  MTBF  is  not  less  than  100  hours  (which 
seems  to  be  a  fairly  high  confidence,  if  the  minimum  requirement  were,  in  fact, 

100  hours).  The  7$  figure  is  usually  referred  to  as  the  Air  Force's  (or  con¬ 
sumer's)  risk.  The  ratio  100/200  -  (abs.  min.  MTBF) /(contractual  MTBF)  -  .5 
is  called  the  discrimination  ratio.  But  suppose  we  haul  chosen  150  hours  Instead, 
giving  150/200  -  (abs.  min.  MTBF) /(contractual  MTBF)  r  .75  as  our  discrimination 
ratio?  Following  the  same  procedure,  with  U  *  1,000/150  -  6.66  (which  must  be 
approximated  by  6.5  in  order  to  use  our  tables)  we  can  be  about  63$  confident 
that  the  true  MTBF  is  not  less  than  150  hours  (which  seems  not  a  very  high  confi¬ 
dence).  In  fact,  the  closer  we  get  to  200  hours  (that  is,  the  higher  our 
discrimination  ratio)  the  lower  U  becomes,  and  the  lower  our  confidence  is.  On 
the  other  hand,  if  the  test  time  had  been  longer,  say  2,000  hours,  and  we  had 
experienced  10  failures,  we  still  get  the  same  computed  MTBF  of  200  hours,  but 
now  (using  the  tables  again,  first  with  U  =  2,000/100,  x  =  10;  and  then  with 
U  •  2,000/150,  x  c  10)  we  can  be  99$  confident  that  the  MTBF  is  not  less  than 
100  hours,  and  77$  confident  that  the  MTBF  is  not  less  than  150  hours.  (The 
reader  should  verify  all  of  these  figures  in  the  tables  to  be  certain  of  his 
understanding  of  the  procedure.)  Thus,  we  see  that  an  increase  in  test  time  can 
give  a  significant  increase  in  confidence. 

3.3  The  Technique  -  The  Producer's  Risk.  Let's  now  look  at  another  question. 
Suppose  operational  requirements  were  such  that  the  device  mentioned  above  had  to 
have  an  MTBF  of  250  hours.  Having  experienced  5  failures  in  1,000  hours  of  test¬ 
ing  should  it  be  rejected?2  (The  computed  MTBF  is  again  200  hours . )  Rephrasing 
again,  what  is  the  probability  of  a  device  having  5  or  more  failures  in  1,000 
hours  -if  it;s  true  MTBF  is  250  hours?  Here  U  :  1,000/250  -  (test  time)/(true 
MTBF)  =  4,  and  reading  across  from  x  -  ^  we  see  that  D(x)  -  37$*  We  may  interpret 
this  as  saying  that  there  is  a  37$  chance  that  good  equipment  is  being  rejected, 
which  seems  rather  high.  (The  figure  37$  is  usually  called  the  producer's  risk.) 


Taak  Group  ]  ot  AGREE  (Adv.aoty  Croup  _>n  Re.ab.l'y  ot  Electrons  Equ  pment)  reported  (June  1957)  that  there  are 
at  lea*'  ?h*ee  possible  meanings  wh  h  can  bt  assoc  a  ed  w  h  in  n. mum  acceptability  t.gtre*  to.-  re.iab:’.ity 

(a)  that  vnlue  wh  eh  the  opera*. onal  commande  wi'l  ole.ate  and  below  wh  li  he  would  take  drastic  action  to  .nitiate 
imp  o^emm's  (b,  tha  value  wh..h  agrees  w  ,h  he  cu  »en.  ’■etitfbi  y  .a  m  •»  observed  tor  each  ctaaa  of  equipment, 
and  {cl  tha.  value  which  the  current  statr.ol  he-art  could  achieve  We  hn^e  m  mind  here  that  the  baolute 

min  mum  MTBF"  could  be  net  in  accordance  w»th  (a)  and  -he  ‘conVar*  MTBE”  could  he  net  in  accordance  wth 

(b)  or  (c). 

^  Tbi«  is  no  the  ptace  to  d.acnaa  the  mean.ng  (0;  consequences;  ..{  accept  «■  rejrrt  decision*  *  nee  we  are 
mostly  interested  in  eiplnm.ng  methodology  lor  reaching  such  decia.ona.  |i  i»  e^per  -ed  howeve*  *hat  we  are 
nor  tatk.ng  abou.  accenting  or  rejec;:ng  the  pa-».  ntar  equ.pmen*(a>  under  le.t.  I^t  rather  ihe  dea’pn  and 'or  macu¬ 
lae. urng  process  (nsofar  aa  re'nb.iity  in  alfected). 
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Again,  if  the  test  time  was  extended  to  2,000  hours  with  10  failures  occurring, 

U  :  2,000/250  -  (test  time)/(true  KTBF?)  -  8,  and  reading  across  from  x  =  10 
gives  D(x)  Z  28$  chance  of  rejecting  good  equipment,  which  is  somewhat  lower; 
that  is,  the  producer’s  risk  also  lessened  as  the  test  time  increased. 

3 .4  A  Technique  to  Keep  "Risks"  and  "Test  Time"  Low.  These  two  risks 
( -tnsumer *s ,  producer’s)  are  present  in  any  accept/rs Ject  criteria  which  based 
on  tbe  number  of  failures  which  occur  during  a  specified  test  time  The  tables 
in  Figure  3  give  a  listing  of  these  risks  for  various  periods  of  test  time  and 
varying  numbers  of  allowable  failures.  In  these  tables  it  is  assumed  that  the 
contractual  MTBF  is  200  hours,  and  "allowable  failures"  means  that  we  accept  If 
the  number  of  failures  is  less  than  or  equal  to  that  number ,  and  reject  otherwise. 
The  AF  risk  indicated  is  based  on  a  5  discrimination  ratio,  that  is,  we  have 
assumed  that  the  absolute  minimum  KTBF  requirement  is  100  hours.  Remember,  in 
this  context,  A$  confidence  mean b  that  there  is  a  1-(A$)  risk  that  equipment 
having  KTEF  :  100  hours  will  pass  the  test  In  other  words,  we  should  not  speak 
of  tbe  confidence  figure  without  also  stating  the  discrimination  ratio  that  is 
being  used.  Now  let’s  look  at  these  tables  to  see  if  we  can  determine  an  optimum 
test  duration  and  an  optimum  number  of  allowable  failures  for  demonstrating 
reliability.  (See  Figure  3)  As  can  be  seen  from  these  tables,  it  is  extremely 
difficult  to  bold  both  the  producer's  risk  and  the  AF  risk  at  low  levels  when  a 
specific  test  time  is  set  before  hand  A  quick  scanning  of  these  tables  shows 
that  if  risks  of  10$  are  required  by  both  parties,  about  2.000  hours  of  test  time 
would  be  necessary  (see  table  (6)  with  l4  failures  allowed) .  If  only  one  model 
is  available  for  testing,  this  means  about  3  months  of  testing  even  if  testing  is 
conducted  24  hours  a  day  (which  is  usually  impossible).  However,  there  are  ways 
of  reducing  the  test  time  considerably,  thereby  giving  significant  savings  in 
time  and  money.  Suppose  tbe  test  time  was  set  at  2,000  hours,  with  14  failures 
allowed,  but  after  800  hours  of  testing  only  4  failures  bad  occurred.  A  glance 
at  table  (3)  shows  that  tbe  AF  could  accept  at  this  point  with  only  9$  risk. 

The  producer* 8  risk  at  this  point  is  37$  but  tbis  figure  is  irrelevant  since  we 
are  not  rejecting  at  this  point  On  tbe  otber  band,  suppose  that  after  600  hours 
5  failures  had  occurred.  The  AF  could  reject  at  this  point  since  table  (2)  shows 
that  the  producer's  risk  is  8$,  Again,  take  note  that  the  corresponding  AF  risk 
is  44$  but  this  figure  is  irrelevant  since  5  failures  in  600  hours  is  not  being 
specified  for  acceptance  purposes  To  sum  up  these  remarks,  it  would  be  nice  if 
these  earlier  acceptance/rejectance  points  were  built  into  the  demonstration  plan. 
Even  using  tbe  limited  information  contained  in  these  six  tables,  we  could  con¬ 
struct  an  acceptance /re J ec tance  plan  as  follows: 


U 


allowable  failures  vs.  risks  for  preselected  test  tikes 


(1)  TEST  TIKE;  400  horn's 

(4)  TEST  TIME:  1.000  hours  

ALLOWABLE 

PRODUCER’S 

AIR 

.  FORCE'S 

ALLOWABLE 

PRODUCER'S 

AIR  FORCE'S 

FAILURES 

RISK* 

RISK** 

FAILURES  

RISK* 

RISK**  

0 

86  % 

2% 

3 

73%  . 

11 

1 

59% 

9% 

4 

55% 

2% 

2 

32%  

24% 

5 

38% 

6< 

3 

14% 

43* 

6 

24% 

13% 

4 

5% 

63* 

7 

13% 

22% 

8 

6% 

33% 

(2)  TEST  TIME:  600  hours 

(5)  TEJT  TIKE; 1  .300  hour  a 

ALLOWABLE 

PRODUCER'S 

AIR  FORCE’S 

ALLOWABLE 

PRODUCER'S 

AIR  FORCE'S 

FAILURES 

RISK* 

RISK** 

FAILURES 

RISK* 

RISK** 

0 

95% 

.2% 

5 

76% 

.2%  

1 

80% 

2% 

6 

62% 

.7% 

2 

57% 

6% 

7 

48* 

2% 

3 

35% 

13% 

8 

34* 

4% 

4 

18% 

28* 

9   

22% 

7% 

3 

_ 8%  _ 

44* _ 

10 

14* 

 12% 

11 

8%  „ 

18% 

(3)  TEST  TIME:  800  hours 

(6)  TEST  TIME;  2,000  hours _ 

ALLOWABLE 

PRODUCER’S 

AIR  FORCE’S 

allowable 

PRODUCER'S 

AIR  FORCE’S 

FAILURES  

RISK* 

RISK** 

FAILURES 

RISK* 

RISK** 

0 

98* 

.03* 

7 

78% 

.07% 

1 

90* 

.3% 

8 

67% 

.2% 

2 

76* 

IS.  ^ 

9 

 54% 

4% 

3 

56% 

4% 

10 

 .  42% 

1% 

.4 .... 

37% 

9% 

11 

30% 

2% 

3  

21% 

19%  

12 

21% 

4% 

  6 

-V%  ... 

31% 

13 

14* 

7% 

7 

5% 

45l 

14 

8% 

10% 

•Producer’s  risk:  The  probability  that  equipment  which  exactly  satisfias  the 
requirement  (200  hours)  will  fail  ths  test. 


••Air  Force’s  risk:  The  probability  that  equip  .*  which  has  MTBF  squal  to  l/2 
tbe  requirement  (100  hour  }  •*  1  pass  the  teat. 


Figure  3 
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TEST  TIME 

REJECT  IF  NUMBER 
OF  FAILURES  IS 
GREATER  THAN  OR 
EQUAL  TO: 

PRODUCER'S 

RISK 

CONTINUE  IF 
NUMBER  OF 
FAILURES  IS 
IN  RANGE 
BELOW 

ACCEPT  IF  NUMBER 
OF  FAILURES  IS 
LESS  THAN  OR 

EQUAL  TO: 

AF 

RISK 

400 

5 

5  i 

2-U 

1 

9  i 

600 

6 

8% 

3-5 

2 

6% 

800 

8 

5% 

5-7 

k 

9% 

1000 

9 

6% 

6-8 

C 

6% 

1500 

11 

8% 

10 

9 

7% 

2000 

15 

8% 

■ 

l*t 

10% 

If  tables  had  been  constructed  for  500  ,  700  ,  900,  HOO-lLOO,  and  1600-1900  hours, 
even  more  acceptance/rejectance  pointB  could  be  incorporated  which  gives  greater 
possibility  of  a  decieion  being  reached  prior  to  2,000  houre . 
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CHAPTER  4 

SEQUENTIAL  TESTING  AND  KIL-R-264'74 


The  ideas  presented  in  the  preceding  section  come  under  the  name  "sequential 
testing*'  and  have  been  studied  for  some  time  so  that  formulas  are  available  to 
select  "certain"  decision  points  beforehand.  (See  Attachment  1.)  These 
formulas  ere  the  basis  for  Die  test  plan  given  in  Table  I  of  MIL-R-26474  and 
use  10%  risk  conditions  (maximum)  end  a  .5  discrimination  ratio."1  The  formulas 
eneble  us  to  stele  test  time  more  generally  -  in  terms  of  multiples  of 
contractual  MTBF.  Ihere  is,  however,  one  major  problem.  Stated  simply,  the 
problem  is  that  the  formulas  do  not  find  the  earliest  decision  points  (with 
10 1  risks)  but  instead  find  eerlier  decision  points  (with  risks  considerably 
lower  then  10%).  The  reason  for  this  is  a  mathematical  issue  and  need  not 
oncern  us  here.  However,  these  formulas  ere  given  in  Attachment  2  wherein 
a  derivation  of  Table  I  is  given.  Table  I  appears  in  Figure  4  (p age  19)  showing 
the  eerlier  decision  points  and  the  associated  risks.  Making  a  comparison 
between  Teble  I  and  the  table  derived  in  the  preceding  section  shows  a  narked 
difference.  (Simply  divide  the  test  times  by  200  to  make  this  comparison.) 
Ensuing  paragraphs  will  concentrate  on  the  effects  of  lable  I  rather  than  the 
ceuses . 

The  important  thine  to  notice  eboui  Teble  I  is  that  4  he  Producer's  Risk 
and  AF  Ris>-  stay  at  about  and  3%  respectively.^  This  rtiakes  the  test  quite 
severe  (a4  least  more  severe  than  the  stated  risk  levels  of  10%  indicate)  and 
only  equipment  which  is  extremely  good  or  extremely  bad  is  likely  to  cause  a 
decision  early  in  the  test  span.  Since  the  main  object  of  sequential  testing 
is  to  permit  lecisions  to  be  reached  early  in  the  test  at  predetermined  risk 
levels,  it  is  rather  puzzling  that  the  earlier  decision  points  have  such  low 
risk  levels.  In  other  words,  Table  I  does  not  accomplish  what  it  apparently 
set  out  'o  do  -  namely,  to  permit  decisions  to  be  reached  early  at  risk  levels 
of  10%.  It  may  permit  early  decisions,  but  it  does  so  at  risk  levels  of  2-3%, 
that  is,  95%  confidence  for  the  AF  and  2%  risk  for  the  producer,  recalling 
that  the  confidence  should  not  be  stated  without  also  rivinr  the  discrimination 
retio,  we  should  state  it  tbus:  Teble  I  in  ectuality  gives  97%  confidence  with 


1 Table  I  was  developed  by  Task  Group  2  of  AGREE.  Their  report  war  issued  by  the 
Office  of  the  Assistant  Secretary  of  Defense  (Research  end  Engineering),  4  Jun  1957. 

^Tbe  Producer's  Risk  is  found  by  setting  U  =  multiples  of  contract  MTBF  and 
finding  b(x)  for  x  =  the  REJECT  figure.  The  AF  Risk  is  found  by  setting  U  =  2 
times  ihe  multiples  of  contract  MTBF  and  finding  C(x)  for  x  =  t  he  ACCEPT  figure. 

To  see  this,  let  T  denote  the  time  units  in  Column  I,  then  since  the  discrimina- 


tion  ratio  is  1/2,  U  =  tire\=  /  -test  time  \= 

/  test  time  \ 

\lrue  MTBF/  (1/2  contract  MTBP/ 

(contract  KTBF/ 
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£  discrimination  ratio  of  5,  that  is,  °7%  confidence  that  the  true  >n bF  is  rot 
less  than  !/«;  the  contractual  requirement. 

Now  let  us  look  at  tha  effects  of  using  the  ebove  mentioned  formulas  for 
sequential  testing,  but  changing  the  risk  levels  end  discrimination  ratio  as 

follows ; 


Risk  levels:  20/f  (rraximum) 

Discrimination  Ratio;  -5 

Attachment  2  gives  the  derivet  ion  of  the  table  which  is  obtained  using  these 
velues  (Table  A,  see  Figure  5).  Looking  at  Table  A,  we  sea  that  although  the 
risk  levels  were  set  at  20£  (meximum),  they  ectualiy  naver  reech  this  level. 

In  fact,  the  risks  stey  at  5-8%.  except  at  the  very  end,  when  they  rapidly 
rise  to  “‘t  for  producer  and  12£%  for  the  Air  Fore a. 3  Bear  in  Bind,  however, 
that  if  sequential  testing  accomplishes  what  its  craetors  claim,  tha  11  %  and 
*  levels  should  rerely  be  reached.  In  tha  cese  of  Tabla  A,  this  seems 
quite  reasonable  since  the  CONTINUE  TEST  region  is  relatively  narrow  (ee 
compared  to  Table  I).  A  graphicel  picture  of  both  Table  I  and  Table  A  which 
shews  the  ACCEPT,  REJECT,  and  CONTINUE  TEST  regions  is  given  in  Figure  6,  and 
illustrates  their  markad  differences.  Teble  A  also  ellows  less  time  to  reach 
the  first  decision  point  (1.38  multiples  of  contract  MTBF  instead  of  3)  and 
less  maximum  test  time  (8  multiples  of  contract  MTBF  instead  of  10.3). 
Essentially,  than,  Table  A  is  still  giving  the  Air  Force  about  9 0$  confidence 
(with  e  .5  diecriminetion  ratio)  at  e  considerable  sevings  in  tast  time  (if 
the  MTBF  requirement  is  high).  Whether  Table  A  should  be  used  in  preference 
to  Teble  I  is  a  question  that  cen  only  be  answered  in  consideration  of  several 
factors  such  as: 

(1)  Number  of  models  available  for  testing; 

(2)  Contrectual  MTBF  requirement;  and 

(3)  Determination  of  risk  levels  that  era  consistant  with  achedulas, 
cost  of  testing,  test  environment ,  end  equipment  characteristics-  (for  example, 
the  amount  of  development  involved ) . 

In  the  casa  of  high  MTBF  requirements ,  thera  era  other  considerations 
which  may  serve  to  Justify  the  increased  risk  levals  of  Tabla  A.  Looking  first 
at  the  increased  risks  to  tha  producer,  consider  that  sinca  tha  Air  Forca  hes 
sacked  away  from  the  contrectual  MTBF  requirement  in  order  to  compute  AF  Risk, 
it  would  not  seem  unreasonable  to  expect  the  producer  to  compute  hie  risk 
based  on  some  higher  value  than  tha  contractual  MTBF  requirement.  Certainly, 
the  producer  cennot  design  the  equipment  MTBF  to  exact  numerical  values.  One 
veu  d  expect  thet  efforts  would  be  directed  toward  designing  somewhat  better 
then  the  requirement.  If  we  arbitrarily  selected  "1  /L  higher”  va  may  usa  the 
Tcicson  tebles  to  show  that  the  producer's  risk  is  quite  low.  These  risks  are 
indicated  in  Table  A.  It  is  quita  aasy  to  justify  the  incraasas  in  AF  Risk 
(especially  whan  the  cost  of  testing  is  $1 , 000  a  day  or  mora)  sinca,  in  the 
esse  of  high  MTBF  requirements ,  there  is  a  considerable  raduction  in  teat 

^  The  po  n*  f<r  end. eg  he  ten*  wan  arhurar.ly  selected  hy  the  techn.qoe  g  ven  •  Chapter  3). 


'-REJECT  CRITERIA  FOR  FAILURE  RATE  TESTING 
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houra  required.  (The  next  chapter  is  eimed  at  making  thia  last  statement  more 
precise.)  But  more  than  test  expenditures  may  be  effected  -  as  time  passes, 
it  becomes  increasingly  harder  to  correct  design  weaknesses,  and  increasingly 
more  costly  to  the  producer.  In  the  final  analysis,  the  Hr  Force  bears  the 
burden  of  both  design  deficiencies,  and  increased  coats  to  the  producer, 
whichever  may  occur  es  e  result  of  delayed  decisions. 

Maximum  teat  time  may  be  shortened  even  further,  but  only  et  the  expense 
of  increesed  risks  to  the  producer  and  consumer.  Once  e  discriminetion  retio 
and  "maxirnim  risks"  are  selected,  it  ia  e  simple  matter  to  scan  the  Poisson 
tables  to  select  a  point  for  ending  the  test  (celled  a  "truncation'  point). 

Thia  technique  was  expleined  in  Chepter  3,  where  a  truncation  point  of  2,000 
hours  (10  multiples  of  MTBF)  wes  selected.  However,  whan  "sequential  test 
formulas"  are  used,  there  are  formulas  which  give  the  first  point  at  which  the 
producer's  risk  ia  equal  to  the  "predetermined"  level  (20%  in  tha  ceae  of  Teble 
A)  and,  the  same  time,  the  consumer's  risk  is  less  than  or  equal  to  the 
predetermined  level*.  If  these  formules  hed  been  applied  to  Table  A,  the 
result  would  heve  been  es  follows  (with  the  new  risks  ss  indiceted): 


1.38 

4 

1-3 

0 

2,07 

5 

2-4 

1 

2.76 

6 

3-5 

2 

3.45 

7 

4-6 

3 

4.U 

7 

i2i * 

5-6 

4 

4.73 

7 

20$ 

- 

17* 

6 

Note  that  the  producer's  risk  climbs  to  2056,  while  the  AF  risk  never  exceeds 
17%.  If  the  producer's  risk  is  computed  on  the  besis  of  designing  higher 
than  the  requirement,  the  lest  two  decision  points  (4.I4  and  4.73  time  units) 
would  hava  risks  of  5%  and  9%  instead  of  12^$  and  20%.  Under  certain  conditions 
thia  plan  may  be  quits  reasonable  for  both  parties. 


4  ’’Trane*  i*<l  lif«  Tmu  1b  Expe»*&tUl  Cbm,”  by  B.  Earn*  in,  Ann*!*  of  M«ib*m«iic*]  Sutintion, 
vol.  23,  333-464. 
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ACCEPT -REJECT  CRITERIA  FOR  FAILURE  RATE  TESTING 
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MTBF  requirement  will  be  rejected.  (Note:  This  is  found  by  setting 
U  =  4/5  times  multiples  of  contract  MTBF  and  finding  D(x)  for  x  =  the 
REJECT  figure.) 


COMPARISON  OF  CONTINUE  TEST 
RED  IONS  OF  TABLE  I  AND  TABLE  A 
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Figure  6 


CHAPTER  5 


AVERAGE  TEST  TIME 


Vhen  the  sequential  test  formulas  (as  given  in  Attachment  Z)  are  used, 
there  are  still  other  formulas  (see  Attachment  3)  which  give  the  most  probable 
number  of  failures  required  for  accept/reject  decision*  if*  the  true  MTBF  of 
the  equipment  is  equal  to  either: 

Tc  :  the  contractual  MIBF  requirement,  or 

Ttt  :  the  "absolute  minimum"  MTBF  requirement 

E(X),  the  most  probable  number  of  failures  required  if  the  true  MTBF  is  X, 
is  calculated  for  each  of  the  values  Tc  and  Tm  in  Attachment  3,  first  for 
Table  I  and  then  for  Table  A.  Comparing  these  values  with  the  corresponding 
tables  we  obtain  the  following  results: 


TABLE  I 

TABLE  A 

Tc* 

6.34 

3.45 

Tm* 

4.01 

1.38 

where  the  figures  shown  are  in  multiples  of  contract  MTBF1  and 

Tc*  =  the  average  time  units  for  acceptance  If  the  true  MTBF  is 
equal  to  Tc  (the  contractual  MTBF  requirement) 

Tm*  -  the  average  time  units  for  rejection  if  the  true  MTBF  is 
equal  to  Ta  (1/2  the  contractual  MTBF  requirement) 

These  figures  show  that,  on  the  average,  testing  via  Table  A  will  be 
completed  in  less  than  half  the  time  required  by  Table  I.  The  impact  of  test 
time  on  the  overall  program  must  nevertheless  be  weighed  against  the  risks 
that  are  involved.  At  any  decision  point,  we  decrease  our  confidence  by 
about  5%  when  using  Table  A  instead  of  Table  I.  In  both  cases  our  confidence 
is  associated  with  having  1/2  the  contractual  MTBF  requirement. 


•  io  octvai  tMi  tiao,  aakiply  tbe  time  ulti  ltd  letted  by  tbc  coeuacttel 


1  To  chtnge  T*  «ad  T, 
MTBF 
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CHAPTER  6 


METHODS  TO  BE  EMPLOYED  IN  FUTURE  PROCUREMENTS 


Thara  ia  raaaon  to  believe  that  the  authors  of  MIL-R-26474  realized  the 
shortcomings  of  Tabla  I,  since  they  provided  exceptions  in  the  event  that 
conditions  required  termination  of  testing  prior  to  reaching  an  accept/re¬ 
ject  decision.  (Saa  Conditions  I  and  II  of  MIL-R-26474* )  Tha  specification 
provides  that  if  3  multiples  of  contract  MTBF  have  been  reached,  and  if 
tha  final  reliability  estimate  ia  "clearly"  greeter  than  the  contractual 
requirement,  than  an  accept  decision  will  be  made.  However,  how  much  greater 
ia  "clearly"  greater,  and  on  what  kind  of  date  should  the  final  reliability 
estimate  be  baaad,  ia  not  spelled  out  in  the  specification.  For  example,  do 
we  completely  ignore  the  results  of  our  (3  multiples  of  MTBF)  tasting  that 
was  accomplished?  A  provision  such  as  this  pre-aupposea  that  tha  procuring 
sgency  was  not  able  to  develop  an  acceptance/rajectance  criteria  that  would 
give  a  substantial  level  of  confidence  for  shorter  periods  of  testing  (auch 
es  Table  A)  To  illustrate  tha  possible  consequences  of  this  condition, 
suppose  that  aftar  3  multiples  of  testing  7  relevant  failures  hsd  occurred, 
and  that  the  contractor's  predictions  were  still  higher  than  the  contractual 
requirement.  Under  MIL-R -26474  the  Air  Force  might  be  forced  to  accept 
the  equipment,  and  et  the  staggering  risk  of  74%;  thet  is,  the  probability 
thet  equipment  heving  \  the  required  MTBF  would  heve  7  or  less  failures 
in  3  multiples  of  contractual  MTBF  is  74%.  Yet,  if  Table  A  was  in  force, 
the  equipment  would  be  rejected  with  less  than  7%  risk  to  the  contractor. 

In  view  of  the  above  mentioned  amhiguities  and/or  the  unwarranted  risks 
that  could  result,  MIL-R-26474  should  not  be  ci+ed  in  future  procurements, 
unless  the  specific  accept/reject  criteria  ia  defined  in  the  equipment 
specification  or  work  statement  giving  the  conditions  thet  apply  if  termi¬ 
nation  of  teBting  becomes  necessary,  and  only  after  full  consideration  of 
the  riaks  that  may  be  involved.  In  addition,  the  work  statement  should 
call  for  e  reliability  demonstration  plan  which  must  clearly  state  the 
ground  rules  for  counting  failuraa  and  measuring  operating  tima,  as  well  as 
the  operating  conditions  that  will  prevail  during  the  demonstration. 
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ATTACHMENTS  1  THROUCH  4 


VERIFICATION  OF  QUANTITATIVE  RELIABILITY  REQUIREMENTS 
decision  Criteria 
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1  *  Underlying  Assumption.  The  elements  of  set  theory,  wherein  e  point 
(element;  is  called  an  "outcome  of  an  experiment",  the  collection  of  ail 
points  (outcomes)  is  called  the  "sample  space"  (and  denoted  by  G),  and  any 
sub-collection  of  points  (outcomee)  of  G  is  called  an  "event".  In  particular 
G  is  an  event,  and  the  collection  consisting  of  no  points  (outcomes)  is  an 
event  (called  the  "null  event"  and  denoted  by  0) .  A  "random  variable"  is  a 
numerically-valued  function  defined  over  the  sample  space  Gj  i.e.,  a  rule  which 
assigns  exactly  one  number  to  eech  outcome. 


2.  Basic  Axioms.  Ve  assume  the  existence  of  e  function  P  satisfying  the 
following  axioms1 . 


e. 

Axiom  1 : 

p{g)  = 

1 

b. 

Axiom  2: 

?{t)  = 

0 

c. 

Axiom  3: 

0  <  p{e  } 

<  1  for  any  event  E 

d. 

Axiom  4: 

p{EiUE2) 

■  =  p{ei)  +  p{e2'\ 

NOTE:  If 

(ei  Ae2| 

=  0  then 

[ei  U  b2\ 

=  p(ei>  +  p(e2) 

3.  Cujculetlve  Distribution  Function.  For  any  random  variable  X  =  X  (w)  the 
function  defined  by 

Fj  :  Fy  (a)  =  P  (w)  <  aj>  where  (-oo<e£o o) 

ie  celled  the  c^mylflt&ve  distribution  function. 


4-  Pro parties, of  the  Cumulative. Distribution  Function.  The  cumulative 
distribution  function  hes  the  following  basic  properties: 


a.  It  is  a  non-decreasing  function. 

b.  Fi  (-  oo)  =  0 

c.  Fi  (oo)  =  1 


1  P  (E) 


r  “Tke  (numerical)  probability  of  the  event  E  occ wring”,  {EjO  Ej]  tignifie*  the 
»  occurrence  of  Ej  end  Ej.  (Ej(J  E2>  ie  *n  even;  which  occurs  when  either  Ej  or  Ej  occwe. 
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derivation  of  table  it  MH-R-^6474 

Teble  I  ie  derived  through  the  use  of  the  following  formulas  which  were 
developed  by  mathematicians  Epstein  and  Scbel1 .  Ve  shall  begin  by  defining: 

OC  =  Producer's  Risk  y  fi  =  AF  Risk 

Tc  =  Contractual  MTBF  requirement 

Tm  =  Absolute  minimum  required  MTBF 

fa  =  Number  of  failures  for  acceptance 

fr  =  Number  of  feilures  for  rejectance 

ta  =  Teet  time  for  ecceptancfl  (ultimately  expreseed  as  a  function  of  fe) 

tp  =  Test  time  for  rejectance  (ultimately  expressed  es  e  function  of  fr) 

In  =  Abbrevietion  for  the  natural  logarithm  function. 


The  equations  vre  need  are* 


Tc  -  Tm 

Tib 


-1 


Tc  -  T„ 


Now,  to  illustrate  the  use  of  these  formulas,  we  first  develop  Teble  I  in 
which. 


(1) 

c<  = 

10$  =  .1 

(2) 

'Ob 

it 

10$  =  .1 

(3) 

Trr.  = 

Is. 

1  See  AGREE  (Advinory  Croup  on  Reliability  erf  Elecjonic  Eqmpmcni)  Report,  4  June  1957.  Sapt  of  Document*, 
U.  S.  Government  Pr.m.ng  Office,  Wash  25,  D»  C. 
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Preliminary  Calculations: 


1  3y  (3),  Tfc  =  2, and  Tc-Tm  -  -j 

Tm  Ta 

2.  By  (1)  and  (2)  and  standard  mathematical  tables  ve  get: 

-'“fer)"  -*(ir  -xr  * 9  *  “ 


*©■ 


In  2  =  .69 


Nov  ve  make  tha  approprieta  subetitutions  giving  us: 

For  Accaptanca:  ta  =  2.2  +  .69  fe 
For  Rejactanca:  tr  =-2.2  +  .69  fr 

Next  ve  simply  eolve  thesa  aquatione  for  numbers  of  failures  =  0,  1 ,  2,-. etc. 


fe 

fa 

fr 

.  v 

0 

2.20 

0 

-  2.20 

1 

2.89 

1 

-  1.51 

2 

3.58 

2 

-  0.82 

3 

4.27 

3 

-  0.13 

4 

4.96 

4 

0.56 

5 

5.65 

5 

1.25 

6 

6.34 

6 

1.94 

7 

7.03 

7 

2.63 

8 

7.72 

8 

3.32 

9 

8.41 

9 

4.01 

10 

9.10 

10 

4.70 

11 

9.79 

11 

5.39 

->2 

10.48 

12 

6.08 

13 

11.17 

13 

6.77 

U 

11.86 

14 

7.46 

15 

12.55 

15 

7.72 

Since  the  aquations  for  acceptance/re jactance  are  straight  lines  vith  tha 
same  slope,  the  above  liete  could  go  on  indefinitely  (never  reaching  a  point 
where  tha  number  of  failurae  for  acceptance  is  one  less  than  tha  number  of 
failures  for  rsjsctance,  for  the  same  test  timej  i.e.,  vith  ta  =  tr. 

Hovever,  it  was  ehovn  in  Chapter  3  that,  as  test  time  is  increased,  ve 
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v 


eventually  reach  a  point  where  thie  occurej  i.a.»  the  riske  are  comparahla 
for  acceptance/rajectance  failures  differing  by  ona.  Epstein^,  in  fact,  hae 
developed  formulas  for  finding  a  truncation  point  for  vhich  =  10%  and 

never  exceeds  10%.  These  formulas*  vhen  applied  to  Tahle  I  give  a  point  of 
truncation  of  10.30  time  units  vith  15  failures  for  rejactance  Thesa  vera 
used  by  tha  writers  of  MIL-R-26474*  but  here  again,  the  formulas  have  a 
limitation  in  that  they  do  not  find  tha  earliest  truncation  point  vith  c<  and 
ft  approximately  equal  to  10%  Tha  earliest  point  would  be  9-5  time  units 
vith  14  failures  specified  for  rejectance.  To  saa  this  more  clearly,  tha 
and  of  Tahle  I  could  have  bean  established  as  follovs,  vith  risks  as  shovnj 


7.46 

u 

2% 

8-13 

7.72 

14 

3% 

9-13 

8 

3% 

8  15 

U 

4 % 

9-13 

8.41 

14 

5% 

10-13 

9 

3% 

9  10 

14 

7% 

11-13 

10 

3% 

9  50 

14 

10% 

13 

10% 

The  authors  of  Tahle  I  also  arbitrarily  decided  that  teeting  less  than  three 
multiples  of  MTBF  is  a  minimum  requirement  (in  order  to  give  aoma  assuranca 
that  equipment  vith  unduly  ehort  life  vill  be  rejected).  Henca  for  f a  =  1 , 
ta  is  changed  from  2.89  to  3 >00  and  a  corresponding  (arhitrary)  allovance  of 
fr  =  8  is  made  at  3-00  time  units  From  hera  on  the  computed  values  ara  used 
until  reaching  the  10.30  point. 

Derivation  of  Tahla  A.  We  define  c<  s  f  Tc,  Tm,  fa,  fr,  ta,  and  tr  as  befora 
and  uaa  the  eama  formulae: 

i.  (  £  ) 

V*. 


^ '‘Truncated  L .fe  Tea'-*  n  'he  E*ponen  ial  Cast",  B.  Epste.n.  Annais  of  Mathema:  cal  SUT-'a'.ica,  Vol  25. 
tf  155-564. 
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tp  =  V  (fr)  1b  (  It  ) 

\o<.  /  T« 


But  this  time, 

0)  «* 

(2) 

(3)  Tb 

PRELIMINARY  CALCULATIONS; 

1,  By  (3),  Tc  =  2  and  Tc-Tm  =  1 

^  ~T~ 

2.  By  (1)  arid  (2)  and  Standard  Mathematical  Tables: 

-In^l-  -ln^1-.^=  -3i^iS^=  -In  4  =-1.386 

ln^Tc^=  In  2  =  .69 

Making  the  appropriate  substitutions: 

Acceptance:  T&  =  1.38  +  .69  fa 
Rejectance:  Tr  =-1.38  +  .69  fr 
Solving  these  for  numbers  of  failures  =  0,  1 ,  2, ..etc. 


Tc-Tm 

Tm 

=  20%  =  .2 
-  20%  =  .2 


fa 

ta 

fr 

0 

1.38 

0 

-1.38 

1 

2.07 

1 

-  .69 

2 

2.76 

0 

3 

3.45 

3 

.69 

4 

4.14 

4 

1 . 38 

5 

4.83 

5 

2.07 
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fa 

ta 

fr 

tr 

6 

5.52 

6 

2.76 

7 

6.21 

7 

3.45 

8 

6.90 

8 

4.14 

9 

7.59 

9 

4.83 

10 

8.28 

10 

5.52 

11 

6.21 

12 

6.90 

An  interesting  festure  of  this  table  is  that  test  times  for  acceptance/ 
rejectance  coincide  (since  the  sbsolute  value  of  the  constant  term  is  twice 
the  slope)  For  this  table  we  do  not  concern  ourselves  with  setting  an 
arbitrary  minimum  test  time,  since  this  table  is  designed  for  high  MTBF 
requirements  wbere  the  number  of  models  available  for  testing  is  limited. 
Hence,  we  start  our  table  st  1.38  time  units.  Now,  if  we  used  the  formulas 
for  truncation  our  table  would  look  as  follows^ 


1.38 

4 

1-3 

2.07 

5 

2-4 

2.76 

6 

3-5 

3.45 

7 

4-6 

4.14 

7 

12j* 

5-6 

4.73 

7 

20* 

- 

0 

1 

2 

3 

4 

6 


with  risks  indicated  However,  we  bave  no  desire  to  let  the  risks  climb  as 
bigb  as  20%  (producer)  and  17%  (AF)  Therefore,  ws  bave  arbitrarily  extended 
our  tsble  to  8  0  time  units.  This  point  was  selected  by  scanning  the  Poisson 
tables  until  reaching  s  point  of  comparsble  risks  whicb  are  close  to  10%.  The 
derived  table  is  included  in  Chapter  4  of  this  pamphlet  showing  the  risks 
levels  (11%  and  12$%,  respectively)  which  are  reached. 
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DERHATIQff_QZ^AYERAGE  _  TEST  TIME  FOR  TABLE  I_ANP_ TABLE  A 


Firat  define 

Tc  =  contract  HTBF  requirement 

Tm  =  eb so lute  minimum  HTBF  requirement 


Tm 

0<.  =  the  producer's  rl^k  (maximum)  used  for  obtaining  sequential  teat 
formulas 

~  the  AF's  risk  (maximum)  used  for  obtaining  sequential  test  formulas. 

Ve  may  now  give  formulas  developed  by  Epstein  and  Sobel^  for  calculating 
E  (X),  the  most  probable  number  of  failures  required  for  e  decision  if  the 
true  HTBF  of  the  equipment  is  X. 

These  formulas  are  aa  follows  (where  "In"  denotes  the  natural  logarithm 
function) : 


e(tc)~ 


(1-«C)  ln(  &  )  +  c<ln  ( 1  -  ) 

1-  oC    oc 


E(Ttt)  ~ 


In  r  -  (r  -  1 ) 

Ain  (_£_)  +  O-fi)  In  (1-1  ) 


In  r 


<*?) 


Let's  calculete  E  (Tc)  and  E  (Tm)  for  Table  I  and  Table  A,  approximating  tc 
whole  numbers  et  the  end  since  our  answer  must  be  "numbers  of  failures'1. 

TABLE  I 

Ve  heve  r  =  ^c  =  2,  oC  =  .  10  p  ft  -  ,10 

Tm 

Since  ln^1_^=  -In  9,  ve  obtain 


*  "Se^ocD:  *]  Life  Te»'j  10  '.be  Exponen  al  C»»e*\  by  B.  Ep**mn  end  M«  Sobe|£  Ann«l»  of  Math.  Stat., 
Mar  l<n5.  pp  82-93. 
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e  (tc)  =  (.9)(-in  2L±-LdK3fl  9)  =  (,9) 

(In  2)-(2-i5  (.7)-1 

=  -1.98  ±  .22 
-.3 

=  u& 

.3 

—  6 

E  (Tm)  *  (t1)  (-^.3)  +  L9)-.(1^93  =  LH  t-Z.iQ  +L3).(2,2l 
“  in  2  -  (H)  (.7)  -  (.5) 

2  =  -.22  +  1.98 

.2 


=  1.76 

.2 


~  9 


-^ABLea 

Ve  have  r  =  Tc  =  2,  oC  =  .20,  £  =  .20 

5k 

and  (1-*)  =  .8,  _g_  =  1  ,  I-/3  =  4- 

1-<*  4  *. 

Since  In  1  =  -In  4,  ve  obtain 
4 

..  (7C'  =  L 31  (-in  4)  ±  1.2)  (In  4)  --  L3lAzl^2L±J^2XJ±l22l 
(In  2)  -  (2-1 5  (.7)  -  1 

=  ^1  112  +  .278 

*•3 

=  *324 
.3 

a  (TB)  =  (^2M-ln_4)  ±. L31  On  4)  =  LUJLz i,39)  -  (■8).^r32l 

In  2  -(2£L)  (  7)  -  (.,) 

=  (-.278)  +  13,1123 

.2 


=  >634  ^4 

.2 
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X 


The  calculated  number  of  failures  can  be  compared  with  Table  I  to  obtain 


Tc*  =  the  average  time  units  for  acceptance  if  the  true  MTBF  is  equal 
to  the  contract  MTBF 

Tm*  =  the  average  time  units  for  rejectance  if*  the  true  MTBF  is  1/2  the 
contract  MTBF 

These  are 

Tc*  =  6.34 
Tm«  =  4.01 

The  calculated  number  of  failures  can  be  compared  with  Table  A  to  obtain 
Tc*  =  3.45 
Tb*  =  1.38 

Now  comparison  of  Tc*  and  T®*  for  Table  I  and  Table  A  is  readily  aeon  by  the 
following  table: 


TABLE  I 

TABLE  A 

Tc* 

6.34 

3.45 

Tm* 

4.01 

1.38 

which  shows  that  average  test  time2  using  Table  A  is  less  than  half  that  of 
Table  I  let,  our  confidence  is  reduced  only  by  about  3-7%  (During  early 
stages  of  testing  acceptance  is  made  with  90%  confidence  instead  of  97%, 
whereas  if  testing  is  carried  to  the  end  of  the  tables,  acceptance  is  made 
with  87j%/confidence  instead  of  91%). 


2 


To  convert 


aaita”  to  "test  tine”  amply  naltiply  the  tine  aaita  by  tbe  coe tract  MTPF  reqaireamaL. 
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TABLE  OF  THE  POISSON  DISTRIBUTION1 


U  =  0.10 


X 

P(x) 

C(x) 

D(X) 

0 

0.905 

0.905 

1.000 

1 

0.090 

0.995 

0.095 

2 

0.005 

1.000 

0.005 

U  =  0.15 


X 

P(X) 

C(x) 

D(X) 

0 

0.861 

0.861 

1.000 

1 

0.129 

0.990 

0.139 

2 

0.010 

0.999 

0.010 

3 

0.000 

1.000 

0.001 

U  =  0.20 


X 

P(X) 

C(x) 

D(X) 

0 

0.819 

0.819 

1 .000 

1 

0.164 

0.982 

0.181 

2 

0.016 

0.999 

0.018 

3 

0.001 

1 .000 

0.001 

U  =  0.25 


X 

P(x) 

c(x) 

D(X) 

0 

0.779 

0.779 

1 .000 

1 

0.195 

0.974 

0.221 

MOTE:  For  certain  appllcetions  of  the  Poisson  Distribution,  veluea  auch 
es  C(2)  =  1.000  would  be  erronaoua  even  if  the  equal  aign  vaa 
changed  to  "approximately  equal"  (  — ).  Tbia  is  a  consequence  of 
rounding  at  three  decimal  placae. 


"’The  table  ves  computed  on  a  Pbilco  2000  computer  (Model  212)  at  ESD  Space 
Track  Facility.  The  program  is  on  file  et  the  OPR.  Saa  page  12  for 
definitions  of  U,  P(X) ,  C(X),  and  D(X). 
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PCX) 


C(X) 


e.024  0.998 

0.002  1.000 


Us  0.30 


X 

PCX) 

CCX) 

0 

0.741 

0.741 

1 

0.222 

0.963 

2 

0.033 

0.996 

3 

0.003 

1.000 

U* 

0.35 

X 

PCX) 

CCX) 

0 

0.705 

0.705 

1 

0.247 

0.951 

2 

0.043 

0.994 

3 

0.005 

1.000 

Us 

0.40 

X 

PCX) 

CCX) 

0 

0.670 

0.670 

1 

0.268 

0.938 

2 

0.054 

U.992 

3 

0.007 

0  .999 

4 

0.001 

1.000 

Us  0.45 

x  PCX)  CCX) 

o  0.638  0.638 
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OCX) 

0.026 

0.002 


OCX) 

1*000 

0.259 

0.037 

0.004 


OCX) 

1.000 

U.295 

0.049 

0.006 


OCX) 

1.000 

0.330 

0.062 

0.008 

0.001 


OCX) 

1.000 


X 

PCX) 

CCX) 

OCX) 

1 

0.287 

0.925 

0.362 

2 

0.065 

0.989 

0.075 

3 

0.010 

0.999 

0.011 

4 

0.001 

1,000 

0.001 

Us 

0.50 

X 

PCX) 

CCX) 

PCX) 

0 

0.607 

0.607 

1.000 

1 

0.303 

0.910 

0.393 

2 

0.076 

0,986 

0.090 

3 

0.013 

0.998 

0.014 

4 

0.002 

1.000 

0.002 

U» 

0.55 

X 

PCX  ) 

CCX) 

PCX) 

0 

0.577 

0.577 

1.000 

1 

0.317 

0,094 

0.423 

2 

0.087 

0,982 

0.106 

3 

0*016 

0.998 

0.018 

4 

0.002 

1.000 

0.002 

U* 

0.60 

X 

PCX) 

CCX) 

PCX) 

0 

0.549 

0.549 

1.000 

1 

0.329 

0,678 

0.451 

2 

0.099 

0.977 

0.122 

3 

0.020 

0,997 

0.023 

4 

0.003 

1,000 

0.003 
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* 

P(X  ) 

C(X) 

D(X) 

0 

0.522 

0.522 

1.000 

1 

0*339 

0.661 

0.47b 

2 

0.HO 

0.972 

0.139 

3 

0.024 

0.996 

0.028 

4 

0.004 

0,999 

0.004 

5 

0.001 

1.000 

0.001 

1* 

0.70 

X 

P<X) 

c<x> 

DU) 

0 

6.497 

0.497 

1.000 

1 

0.348 

0.844 

0.503 

2 

0.122 

0.966 

0.156 

3 

0.028 

0.994 

0.034 

4 

0.005 

0.999 

0.006 

5 

0.001 

1.000 

0.001 

la 

0.75 

X 

P<X) 

cm 

OCX) 

0 

6.472 

0,472 

1.000 

1 

6.354 

0.827 

0.528 

2 

0*133 

0.959 

0.173 

3 

0.033 

0.993 

0.041 

4 

0*006 

0.999 

0.007 

5 

0.001 

1.000 

0.001 

la 

0.80 

X 

PtX) 

cm 

DU) 

0 

6.449 

0,449 

1.000 

1 

6.359 

0.809 

0.551 

2 

0.144 

0.953 

0.191 

3 

0  •  038 

0.991 

0,047 
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X 

PtX  > 

cm 

D  t  X ) 

4 

0.008 

0,999 

0.009 

5 

0.001 

1.000 

0.001 

Ua 

0.85 

X 

PtX) 

cm 

DtX) 

0 

0.427 

0.427 

1.000 

1 

0.363 

0.791 

0*573 

2 

0.154 

0.945 

0.209 

3 

0.044 

0.989 

0.055 

4 

0  0  0  0 

0.998 

0,011 

5 

n 

1 ,000 

0  ,UL  2 

Ua 

0.90 

X 

PtX  ) 

CtX  ) 

DtX  ) 

0 

0.407 

0.407 

1.000 

1 

0.366 

0.772 

0.593 

2 

0.165 

0.937 

0.228 

3 

0.049 

0.987 

0.063 

4 

0.011 

0.998 

0.013 

5 

0.002 

1.000 

0.002 

Ua 

0.95 

X 

PtX) 

cm 

DtX) 

0 

0.387 

0.387 

1.000 

1 

0.367 

0,754 

0.613 

2 

0.175 

0.929 

0.246 

3 

0.055 

0,984 

0.071 

4 

0.013 

0,997 

0.016 

5 

0.002 

1.000 

0.003 
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US 


u*  1*00 


PCX) 

CCX) 

DCX) 

0.368 

0,366 

1.000 

0.368 

0.736 

0.632 

0.184 

0.920 

0*264 

0.061 

0.981 

0.080 

0*015 

0.996 

0.019 

0*003 

0*999 

0.004 

0*001 

1*000 

0*001 

Us  1.05 
X  P<X) 

o  0*350 

1  0#  367 

2  0.193 

3  0*068 

4  0*018 

5  0*004 

6  0*001 


c<xi  dixj 

0*350  1*000 

0,717  0.650 

0.910  0.283 

0.978  0.090 

0,996  0.022 

0.999  0.004 

1,000  0.001 


U«  1.10 

x  pm 

o  U.333 

1  1.366 

7  *201 

3  0.074 

4  0*020 

5  0.004 

6  0.001 


CCX) 

DCX) 

0*333 

1*000 

0.699 

0*667 

0*900 

0*301 

0.974 

0*100 

0.995 

0*026 

0*999 

0*005 

1.000 

0*001 

U>  1.15 

x  pcx)  cm  Dm 

o  0.317  0.317  1.000 
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X 

PCX) 

CCX) 

OCX) 

1 

Q.364 

0*661 

0*683 

2 

0-209 

0.690 

0.319 

3 

0.080 

0,970 

0.110 

4 

0*023 

0,993 

0*030 

5 

0.005 

0*999 

0*007 

6 

0*001 

1*000 

0*001 

U« 

1.20 

X 

PCX) 

CCX) 

DCX) 

0 

0*301 

0.301 

1.000 

1 

0*361 

0*663 

0.699 

2 

0*217 

0.879 

0.337 

3 

0.087 

0.966 

0.121 

4 

0.026 

0,992 

0.034 

5 

0*006 

0.998 

0.008 

6 

0.001 

1.000 

0.002 

U* 

1.25 

X 

PCX) 

CCX) 

DCX) 

0 

0*287 

0.267 

1.090 

1 

0.358 

0,645 

0.713 

2 

0.224 

0.668 

0.355 

3 

0*093 

0.962 

0*132 

4 

0.029 

0,991 

0.038 

5 

0.007 

0*998 

0*009 

6 

0*002 

1*000 

0.002 

U-  1*30 


X 

PCX) 

CCX) 

DCX) 

0 

0*273 

0.273 

1.000 

1 

0.354 

0*627 

0.727 

2 

0.230 

0.857 

0.373 

3 

0.100 

0.957 

0*143 
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X 

MX) 

cm 

D(X) 

4 

0.032 

0.989 

0.U43 

5 

0.008 

0.998 

0.011 

6 

0.002 

1.000 

0.002 

u  * 

1.35 

X 

P(X> 

cm 

D  <  X ) 

0 

0.259 

0.259 

1.000 

1 

0.350 

0.609 

0.741 

2 

0.236 

0.845 

0.391 

3 

0.106 

0,952 

0.155 

4 

0 . 036 

0.966 

0.046 

5 

0.010 

0.997 

0.012 

6 

0.002 

0.999 

0.003 

7 

0.000 

1.000 

0.001 

U* 

1.40 

X 

p<xi 

cm 

om 

0 

0.247 

0.247 

1.000 

1 

0.345 

0,592 

0.753 

2 

0.242 

0.833 

0.408 

3 

0  .113 

0,946 

0.167 

4 

0. 039 

0.986 

0.054 

5 

0.011 

0,997 

0.014 

6 

0.003 

0,999 

0.003 

7 

0.001 

1.000 

0.001 

u  ■ 

1.45 

x 

pm 

cm 

om 

0 

0.235 

0.235 

1.000 

1 

0.340 

0.575 

0.765 

2 

0.247 

0.621 

0.426 

3 

0.119 

0,940 

0.179 

4 

0.043 

0.984 

0.060 
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X 

MX) 

cm 

om 

5 

0.013 

0.996 

0.016 

6 

0.003 

0,999 

0.004 

7 

0.001 

1,000 

0.001 

lie 

1.50 

X 

P(X) 

cm 

D  <  X  ) 

0 

0.223 

0,223 

l.n00 

1 

0.335 

0,556 

0.777 

2 

0.251 

0.809 

0.442 

3 

0*126 

0,934 

0.191 

4 

0.047 

0,981 

0.066 

5 

0.014 

0,996 

0.019 

6 

0*004 

0.999 

0.004 

7 

0.001 

1.000 

0.001 

u  « 

1.55 

X 

pm 

cm 

om 

0 

0.21? 

0.212 

1.000 

1 

0. 329 

0,541 

0.788 

2 

0.255 

0.796 

0.459 

3 

0.132 

0.926 

0.204 

4 

0.051 

0.979 

0.072 

5 

0.016 

0,995 

0.021 

6 

0.004 

0,999 

0.005 

7 

0.001 

1.000 

0.001 

lie 

1.60 

X 

pm 

ctx> 

Dm 

0 

0.202 

0.202 

1.000 

1 

0.323 

0.525 

0.796 

2 

0.258 

0.783 

0.475 

3 

0.136 

0.921 

0.217 

4 

0.055 

0,976 

0.079 
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X 

PCX) 

C(X> 

OCX  ) 

5 

U  §  018 

0.994 

0.024 

fc 

0.005 

0.999 

0.006 

7 

0.001 

1.000 

0.001 

u* 

1.65 

X 

PCX) 

CCX) 

DCX) 

0 

0.192 

0.192 

1.000 

1 

0.317 

0.509 

D.808 

2 

0 . 261 

0 « 770 

0.491 

3 

0.144 

0.914 

0.230 

4 

0.059 

0.973 

0.086 

5 

0.020 

0.993 

0.027 

6 

0.005 

0.998 

0.007 

7 

0.001 

1.000 

0.002 

U« 

1.70 

X 

PCX) 

CCX) 

DCX) 

0 

0.183 

0.183 

1.000 

1 

0.311 

0.493 

0.817 

2 

0.264 

0.757 

0.507 

3 

0.150 

0.907 

0.243 

4 

0.064 

0.970 

0.093 

5 

0.022 

0.992 

0.030 

6 

0.006 

0,998 

0.008 

7 

0.001 

1.000 

0.002 

U* 

1.75 

X 

PCX) 

C(X) 

D<X) 

0 

0.174 

0.174 

1.000 

1 

0.304 

0.478 

0.826 

2 

0.266 

0.744 

0.522 

3 

0.155 

0.899 

0.256 

4 

0.066 

0,967 

0.101 
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X 

PCX) 

CCX) 

OCX) 

5 

0.024 

0.991 

0.033 

6 

0.007 

0.998 

0.009 

7 

0.002 

1.000 

0.002 

u* 

1.80 

X 

PCX) 

CCX) 

OCX) 

0 

0.165 

0.165 

1.000 

1 

0.296 

0.463 

0.835 

2 

0.268 

0.731 

0.537 

3 

0.161 

0.891 

0.269 

4 

0.072 

0.964 

0.109 

5 

0.026 

0.990 

0.036 

6 

0.006 

0.997 

0.010 

7 

0.002 

0.999 

0.003 

6 

0.000 

1.000 

0.001 

u« 

1.85 

X 

PCX) 

CCX) 

OCX) 

0 

0.157 

0.157 

1.000 

1 

0.291 

0.448 

0.843 

2 

0.269 

0.717 

0.552 

3 

0.166 

0.663 

0.283 

4 

0.077 

0.960 

0.117 

5 

0.026 

0*988 

0*040 

6 

0.009 

0.997 

0-*  012 

7 

0*002 

0.999 

0.003 

8 

0.001 

1.860 

0.081 

U* 

1.90 

X 

PCX) 

CCX) 

DCX) 

0 

0.150 

0.150 

1.000 

1 

0*284 

0.434 

0.850 

2 

0.270 

0.704 

0.566 
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ar*. 


X 

9IX) 

C  <  X  I 

om 

3 

0.171 

0.875 

0.296 

4 

0.081 

0,956 

0.125 

K 

0.031 

0.987 

0.044 

6 

U  .010 

0,997 

0.013 

7 

0.003 

0.999 

0.003 

8 

0.001 

1,000 

0.001 

U» 

1.95 

X 

PU1 

cm 

om 

0 

0*142 

0.142 

1.000 

1 

0.277 

0.420 

0,858 

2 

0*270 

0,690 

0.580 

3 

5.176 

0.666 

0,310 

4 

1.086 

0,952 

0*134 

5 

0*033 

0.965 

0*046 

6 

0*011 

0,996 

0.015 

7 

0.003 

0.999 

0.004 

8 

0.001 

1.000 

0.001 

U* 

2.00 

X 

PIX) 

C  ( X I 

om 

0 

0.135 

0.135 

1.000 

1 

0.271 

0,406 

0.869 

2 

0.271 

0,677 

0.594 

3 

0.180 

3,857 

0.323 

4 

0.090 

0,947 

0.143 

5 

0.036 

0,903 

0.053 

6 

0.012 

0.995 

0*017 

7 

0*003 

0,999 

0.005 

8 

o.ooi 

1.000 

0*001 
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U«  2.10 


X 

PI  X  ) 

CIX) 

om 

0 

0.122 

0.122 

1,000 

1 

0.257 

0,380 

0.878 

2 

0.270 

0,650 

0.620 

3 

0.189 

0.839 

0,350 

4 

0.099 

0,938 

0.161 

5 

0.042 

0.980 

0.062 

6 

U.015 

0.994 

0*020 

7 

0.004 

0,999 

0.006 

8 

0*001 

1,000 

0.001 

U«  2.20 


X 

pm 

CIX) 

D(X) 

0 

0.111 

0.111 

1.000 

1 

0.244 

0,355 

0,889 

2 

0.268 

0,623 

0.645 

3 

U.197 

0,819 

0.377 

4 

0.106 

0,928 

0*181 

5 

U  •  048 

0,975 

0.0/2 

6 

0.017 

0,993 

0.025 

7 

0.005 

0,998 

0.007 

8 

0.002 

1,000 

0*002 

IB 

2.30 

X 

pm 

cm 

DU) 

0 

0*100 

0,100 

1,000 

1 

0*231 

0.331 

0,900 

2 

0*265 

0.596 

0,669 

3 

0.203 

0.799 

0,404 

4 

0*117 

0,916 

0,201 

5 

0*054 

0,970 

0.084 

6 

0*021 

0,991 

0,030 

7 

0*007 

0.997 

0,009 

8 

0*002 

0  .999 

0*003 

9 

0.000 

1,000 

0,001 
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Us  2.40 


X 

PCX  ) 

c  i  x ) 

D  (  X  ) 

0 

U  •  091 

0  .  t)  91 

1.000 

1 

0.21b 

0,300 

0  .909 

2 

0.261 

u ,  570 

0,692 

3 

0.209 

0,779 

0.430 

4 

0.125 

0.904 

0.221 

5 

0 , 060 

0,964 

0.096 

6 

0.024 

0.900 

0,036 

7 

0 .000 

0,997 

0.012 

0 

0.002 

0.999 

0 .003 

9 

0.001 

1,000 

0.001 

l  = 

2.50 

X 

PCX) 

CCX) 

DCX) 

0 

0.002 

0.002 

1,000 

1 

0.205 

0.287 

0.918 

2 

0.257 

0,544 

0.713 

3 

0.214 

0,758 

0.456 

4 

0.134 

0.891 

0.242 

5 

0.067 

0,958 

0.10V 

6 

0.020 

0.906 

0.042 

7 

0.010 

0,996 

0.014 

ft 

0.003 

0,999 

0.004 

9 

0.001 

1,000 

0.001 

U» 

2.60 

X 

PCX) 

CCX) 

DCX) 

0 

0.0/4 

0,074 

1,000 

1 

0.193 

0,267 

0.926 

2 

0.251 

0,518 

0,733 

3 

0.210 

0.736 

0.482 

4 

0.141 

0,877 

0.264 

5 

0.074 

0.951 

0.123 

6 

0.032 

U.983 

0.049 

7 

0.012 

0,995 

0.017 

e 

0.004 

0,999 

0.005 

9 

O.  001 

1  OOO 

O.OOI 
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D=  2.70 


X 

p(x) 

c(x) 

DU) 

0 

0.067 

0.067 

1.000 

1 

0.181 

0.249 

0.933 

2 

0.245 

0.494 

0.751 

3 

0.220 

0.714 

0.506 

4 

0.149 

0.863 

0.286 

5 

o.oco 

0.943 

0.137 

6 

0.036 

0.979 

0.057 

7 

0.014 

0.993 

0.021 

S 

0.005 

0.998 

0.007 

9 

O.OOI 

0.999 

0.002 

10 

0.000 

1.000 

O.OOI 

U=  2.80 


X 

P(x) 

C(x) 

D(I) 

0 

0.061 

0.061 

1.000 

1 

0.170 

0.231 

0.939 

2 

0.238 

0.469 

0.769 

3 

0.222 

0.692 

0.531 

4 

0.156 

0.848 

0.308 

5 

0.087 

0.935 

0.152 

6 

0.041 

0.976 

0.065 

7 

0.016 

0.992 

0.024 

8 

0.006 

0.998 

0.008 

9 

0.002 

0.999 

0.002 

10 

O.OCO 

1 .000 

0.001 

U=  2.90 


P(x) 

C(x) 

D(X) 

0.055 

0.055 

1.000 

0.160 

0.215 

0.945 

0.231 

0.446 

0.785 

0.224 

0.670 

0.554 

0.162 

0.832 

0.330 

0.094 

0.926 

0.168 

0.045 

0.971 

0.074 

0.019 

0.990 

0.029 
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X 

P(X  ) 

C(X  ) 

DiX) 

8 

0 .007 

0.997 

0.U1U 

9 

0.002 

U  .999 

L  .003 

10 

0.001 

1  .OOU 

U.UU1 

Is  3.00 


X 

P<X> 

C(X) 

UU) 

0 

0.050 

0.050 

1  .uuu 

1 

0.149 

U  .199 

0.950 

2 

0.224 

0.423 

0.801 

\ 

0.224 

0.647 

0.577 

4 

0.168 

0.815 

0.353 

5 

0.101 

0.916 

0.165 

6 

0.050 

0.966 

0.064 

7 

0 .022 

0.988 

0.034 

e 

U  >008 

0.996 

0*012 

9 

0.003 

0.999 

0 .004 

10 

0.001 

1 .000 

0  .001 

U=  3.10 


X 

P(X) 

C(X  ) 

Dm 

n 

0.045 

0,045 

1.000 

1 

0.140 

0.185 

0.955 

2 

0.216 

0.401 

0.815 

3 

0.224 

0.625 

0.599 

4 

0.173 

0.798 

0.375 

c. 

0.107 

0.906 

0.202 

6 

0.056 

0,961 

0 , 094 

/ 

0.025 

0 .9 Pit 

0 .039 

8 

0.010 

0.995 

0.014 

9 

0 .003 

0 ,999 

0 .005 

10 

0 . 001 

1.000 

0.U01 
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U*  3.20 


X 

P(X) 

cm 

0 

U.041 

0.041 

1 

0.130 

0.171 

2 

0.209 

0.380 

3 

0.223 

0.603 

4 

0.178 

0.781 

5 

0.114 

0,895 

6 

0.061 

0,955 

7 

0.028 

0.983 

8 

0.011 

0,994 

9 

0.004 

0,998 

10 

0.001 

1.000 

U»  3.30 


X 

P(X) 

cm 

0 

0.037 

0.037 

1 

0.122 

0.159 

2 

0.201 

0.359 

3 

0.221 

0.580 

4 

0*182 

0.763 

5 

U.120 

0.883 

6 

0.066 

0,949 

7 

0.031 

0.980 

e 

0.013 

0,993 

9 

0.005 

0.998 

10 

0.002 

0.999 

U 

0.000 
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X 

P,x> 

C(X) 

D(X) 

4 

0.001 

0.001 

lc  000 

5 

0.003 

0 .004 

0.999 

6 

0 .007 

0.010 

0.996 

7 

0  .  013 

0.024 

0.990 

e 

0 . 024 

0.0  '6 

0.976 

9 

0.039 

0.088 

0,952 

10 

0.057 

0,145 

0.912 

11 

0 .075 

0.220 

0.855 

12 

0 .091 

'>•311 

0 . 78o 

13 

0.101 

0,413 

0 . 609 

14 

0.105 

0.518 

0.587 

15 

0.102 

0.619 

0.482 

16 

0  .  092 

0,711 

0.381 

1  7 

0.  079 

0,790 

0.289 

18 

0 .063 

0.853 

0.210 

39 

0.048 

0.901 

0.147 

20 

0.035 

0.936 

0 . 099 

21 

0.024 

0.9* 

0.064 

22 

0.016 

0  ,  V  /w 

0.040 

23 

0 . 010 

0,986 

0.024 

24 

0  .  006 

0,992 

0,014 

25 

e.  004 

0.996 

0.008 

2  6 

0.002 

0.998 

0 .004 

27 

0.001 

0,999 

0.002 

28 

0 .001 

0 , 99V 

0.001 

29 

0.  QUO 

1.000 

0.001 
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u*  15.00 


X 

PtX  > 

C<  X  ) 

D  ( X ) 

4 

0.001 

0.001 

1.000 

5 

0.002 

0.003 

0,999 

6 

0.005 

0,008 

0,997 

7 

0p010 

0.018 

0  a  992 

8 

0.019 

0.037 

0  3  982 

9 

0.032 

0.070 

0.963 

10 

0.049 

0.118 

0  a  930 

11 

8,066 

0,185 

0.882 

12 

0.083 

0.268 

0.815 

13 

0.096 

0.363 

0.732 

14 

0.102 

0.466 

0.637 

15 

0,102 

0,568 

0.534 

16 

0-096 

0.664 

0.432 

17 

0.085 

0,749 

0.336 

18 

0.071 

0.819 

0.251 

19 

0.056 

0.875 

0.181 

20 

8.042 

0.917 

0,125 

21 

0,030 

0,947 

0.083 

22 

8.020 

0.967 

0.053 

23 

0.013 

0.981 

0.033 

24 

0,008 

0.989 

0.019 

25 

0.005 

0,994 

0.011 

26 

0.003 

0.997 

0.006 

27 

0  0  02 

0,998 

0.003 

28 

0.001 

0,999 

0.002 

29 

0  0  0  0 

1.000 

0,001 

u*  15.50 


X 

P(X> 

C(X) 

D(X) 

5 

0.001 

0.002 

0.999 

6 

0.004 

0,006 

0,998 

7 

0.008 

0.013 

0,994 

8 

0.015 

0  ,  029 

0.987 

9 

0  026 

0,055 

0.971 

10 

0.041 

0.096 

0.945 

11 

0.058 

0.154 

0.904 

12 

0  074 

0,228 

0.846 

13 

0.089 

0.317 

0.772 

14 

0.098 

0,415 

0.683 
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X 

r  (x> 

cm 

D(X) 

15 

0.102 

0.517 

0.565 

16 

0,096 

0,615 

0.483 

17 

o.ovo 

0.705 

0.365 

16 

0.0/7 

0,762 

0.295 

19 

0.063 

0.646 

0.216 

20 

0.049 

0.694 

0.154 

21 

0.036 

0,930 

0.106 

22 

0.025 

0,956 

0.070 

23 

0  .017 

0.973 

0.044 

24 

0.011 

0.984 

0.027 

25 

0.007 

0,991 

0,016 

26 

0.004 

0,995 

0,00V 

27 

0.002 

0,997 

0.005 

26 

0.001 

0,999 

0.003 

29 

0.001 

0,999 

0.001 

30 

0.000 

1.000 

0.001 

U*  16,00 


X 

P(X) 

cm 

D<X> 

5 

0.001 

0,001 

1,000 

6 

0.003 

0,004 

0.999 

7 

0.006 

0,010 

0.996 

6 

0.012 

0.022 

0.990 

9 

0.021 

0,043 

0.976 

10 

0.034 

0,077 

0.957 

11 

0.050 

0.127 

0^923 

12 

Q.066 

0,193 

0.673 

13 

0.061 

0,275 

0.607 

14 

0.093 

0,368 

0.725 

15 

0.099 

0,467 

.  0,632 

16 

0.099 

0,566 

0.533 

17 

0.093 

0.659 

0.434 

18 

0.063 

0,742 

0.341 

19 

0,070 

0,612 

0.25O 

20 

0.056 

0,666 

0.166 

21 

0.043 

0,911 

0.132 

22 

0.031 

0,942 

0 ,06V 

23 

0*022 

0,963 

0.056 

24 

0.014 

0,978 

0,037 

25 

0.009 

0,967 

U  *  022 

26 

0.006 

0.993 

0.013 

27 

0.  003 

0  ,°96 

0.007 
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X 

P(X> 

cm 

p(X  ) 

28 

0.002 

0  .996 

0,004 

29 

0.001 

0.999 

0,002 

30 

0.001 

0.999 

0 ,001 

31 

0,000 

1.000 

U  ,001 

u« 

16.50 

X 

pm 

cm 

om 

5 

|,001 

0,001 

1,000 

6 

0*002 

0,003 

0,999 

7 

0.005 

0.007 

0.997 

6 

0*009 

0,017 

0.993 

9 

0.017 

0.034 

0,963 

10 

0.026 

0.062 

0.966 

11 

0*042 

0,104 

0,936 

12 

0.056 

0,162 

0.696 

13 

0.074 

0.236 

0,636 

14 

0.067 

0,323 

0.764 

15 

0.095 

0,416 

0.677 

16 

0.098 

0.516 

0.562 

17 

0.096 

0.612 

0.464 

16 

0.066 

0,700 

0,366 

19 

0.076 

0.776 

0,300 

20 

0.063 

0,636 

0.224 

21 

0.049 

0,666 

0,162 

22 

0.037 

0,925 

0.112 

23 

0.027 

0.951 

0.075 

24 

0.016 

0,970 

0,049 

25 

0.012 

0,962 

0,030 

26 

i.006 

0,969 

0.016 

27 

0.605 

0,994 

6.611 

26 

0*603 

0,997 

6,066 

29 

6*662 

6,996 

6.663 

SO 

6.001 

0,969 

6.662 

31 

0.000 

1.666 

6.661 
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U*  17. UO 


X 

P(X> 

C(X  > 

Dl  X) 

6 

0.001 

0,002 

0  .  V99 

7 

0.003 

0,00? 

U,  V9tt 

c 

0.007 

0.013 

U.  V95 

9 

0.014 

0,026 

0.96/ 

10 

0.023 

0.049 

0.9/4 

11 

0.036 

0.085 

0.951 

12 

0  .  0?U 

0.135 

0.915 

13 

0.066 

0.201 

0.865 

14 

0.060 

0.281 

0.799 

15 

0.091 

0,371 

0.719 

16 

0.096 

0.466 

0.629 

17 

0 .096 

0,564 

0.532 

18 

0.091 

0.655 

0.436 

19 

0.081 

0,736 

0.345 

20 

0.069 

0.605 

0.264 

21 

0.056 

0,661 

0.195 

22 

0.043 

0,905 

0.139 

23 

0  •  032 

0,937 

0.095 

24 

0.023 

0.959 

0.U63 

25 

0.015 

0,975 

0.04* 

26 

0.010 

0.985 

0  .  U25 

27 

0.006 

0.991 

0.015 

28 

0.004 

0.995 

0.009 

29 

0.002 

0,997 

0.005 

30 

0*001 

0,999 

0.003 

31 

0.001 

0.999 

0.001 

32 

0*000 

1.000 

0.001 

u* 

17.50 

X 

P(X) 

CCX) 

D(X) 

6 

0.001 

0.001 

l.UOO 

7 

0.003 

0,004 

0.999 

8 

0*005 

0,009 

0.996 

9 

0*011 

0,020 

0.991 

10 

0*019 

0.039 

0.980 

11 

0.030 

0 ,06JB 

0.961 

12 

0.043 

0.112 

0.932 

13 

0.058 

0.170 

0.688 

14 

0.073 

0.243 

O.  83  0 
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X 

MX) 

C(X> 

15 

0.Qb5 

0.328 

16 

0.093 

0.420 

17 

0.096 

0.516 

18 

0.093 

0,609 

19 

0.086 

0.695 

20 

0,075 

0.769 

21 

0.062 

0.632 

22 

0*050 

0,662 

23 

0.038 

0,919 

24 

0.028 

0.947 

25 

0*019 

0.966 

26 

0*013 

0.979 

27 

0*008 

0.967 

28 

0*005 

0.993 

29 

0*003 

0.996 

30 

0*002 

0,996 

31 

0*001 

0,999 

32 

0.001 

0,999 

33 

0*000 

1.000 

u*  18.00 


X 

P(X> 

C(X> 

6 

0 .001 

0.001 

7 

0.002 

0.003 

8 

0.004 

0.007 

9 

0.008 

0,015 

10 

0.015 

0.030 

11 

0.025 

0.055 

12 

0.037 

0,092 

13 

0.051 

0,143 

14 

0.065 

0.206 

15 

0*079 

0.267 

16 

0.088 

0,375 

17 

0*094 

0.469 

18 

0.094 

0.562 

19 

0.069 

0.651 

20 

0 .080 

0.731 

21 

0.066 

0,799 

22 

0.056 

0.855 

23 

0.044 

0,899 

24 

0.033 

0.932 

25 

0.024  . 

0.955 

D<X  > 

0,757 

0.672 

0,580 

0,464 

0.391 

0,305 

0,231 

0,166 

0,118 

0.061 

0.053 

0.034 

0,021 

0,013 

0,007 

0.004 

0.002 

0.001 

0.001 


D(X) 

1.000 

0.999 

0.997 

0.993 

0.965 

0.970 

0.945 

0.906 

0.657 

0.792 

0.713 

0.625 

0.531 

0.436 

0.349 

0.269 

0.201 

0.145 

0.101 

0.068 
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X 

P(X  ) 

C(X  ) 

D(X> 

26 

0.016 

0.972 

0.045 

27 

0.011 

0.983 

0.028 

28 

0*00/ 

0.990 

o;ui7 

29 

0*004 

U  .994 

0*010 

30 

0.003 

0.997 

0.006 

31 

0.002 

0.996 

0.003 

32 

0.001 

0.999 

0.002 

33 

U  .000 

1.000 

0.001 

U=  18, 

,50 

X 

P  (  X  ) 

cu> 

DU ) 

6 

0.001 

0.001 

1,000 

7 

0.001 

0.002 

0,999 

8 

0*003 

0.005 

0.998 

9 

0.006 

0.012 

0,995 

10 

0.012 

0,024 

0.968 

11 

0 .020 

0.044 

0.976 

12 

0.031 

0.075 

0.956 

13 

0.044 

0.119 

0.925 

14 

0.058 

0.177 

0.861 

15 

0.072 

0.249 

0.823 

16 

0.083 

0.332 

0.751 

17 

0.090 

0.423 

0.668 

18 

0.093 

0,516 

0.577 

19 

Q.0V1 

0.606 

Q.484 

20 

0.084 

0.690 

0.394 

21 

0.074 

0.764 

0.310 

22 

0.062 

0.626 

0.236 

23 

0.050 

0.875 

0.174 

24 

0*038 

0.914 

0.125 

25 

0.028 

0.942 

0.086 

26 

0.020 

0.963 

0 . 058 

27 

0.014 

0,977 

0*037 

28 

0.009 

0.986 

0.023 

29 

0 .006 

0  ,992 

0  .014 

30 

0.004 

0.995 

U.UUB 

31 

0.002 

0,997 

0  .  U  05 

32 

0.001 

0,999 

0.  U03 

33 

0.001 

0,999 

G.UU1 

34 

0.000 

1,000 

G.UQ1 
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U*  19.00 


X 

P  <X ) 

cu> 

7 

0.0U1 

0.002 

8 

0.002 

0.004 

9 

6.005 

0,009 

10 

0.009 

0,016 

11 

0.016 

0,035 

12 

0.026 

U ,  061 

13 

0.036 

0,098 

14 

0.051 

0,150 

15 

0.065 

0,215 

16 

0.077 

0,292 

17 

Or  086 

0,376 

18 

0.091 

0.469 

19 

0.091 

0.561 

20 

0.067 

0,647 

21 

6.078 

0.725 

22 

0.068 

0,793 

23 

0.056 

0.649 

24 

0.044 

0,693 

25 

6.034 

0,927 

26 

0.025 

0.951 

27 

0.017 

0,969 

28 

6.012 

0.960 

29 

6.006 

0.986 

30 

6.005 

0.993 

31 

6*003 

0.996 

32 

6.002 

0.996 

33 

0.001 

0,999 

34 

0.001 

0,999 

35 

0.000 

1.000 

U»  19.50 


X 

PUI 

cu> 

7 

6.001 

0,001 

8 

6.002 

0,003 

9 

6.004 

0,007 

10 

6*007 

0.014 

11 

6*013 

0.027 

12 

6.021 

.  0,049 

13 

6.032 

0,061 

D<X) 

0.999 

0.998 

0.996 

0.991 

U.982 

0.965 

0.V3V 

0.902 

0.850 

0.765 

0.706 

0.622 

0.631 

0.439 

0.353 

0.275 

0.207 

0.151 

0.107 

0.073 

0.049 

0.031 

0.020 

0.012 

0.007 

0.004 

0.002 

0.001 

0.001 


D<X) 

1.000 

0.999 

0.997 

0.993 

0.966 

0.973 

0.951 
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X 

MX) 

cm 

D<X) 

14 

0.045 

0.126 

0.919 

15 

0 . 058 

0.184 

0 .8/4 

16 

0.071 

0.255 

0.616 

17 

0.081 

0.336 

0.745 

18 

0.088 

0.425 

0.664 

19 

0.091 

0.515 

0.575 

20 

0.088 

0.603 

0.465 

21 

0.082 

0.685 

0.39/ 

22 

0.073 

0.758 

0.315 

23 

0 .062 

0.820 

0.242 

24 

0.050 

0.870 

0.180 

25 

0.039 

0.909 

0.130 

26 

0.029 

0.938 

0.091 

27 

0.0X1 

0.959 

0.062 

?e 

0  .015 

0.974 

0.041 

29 

0.010 

0.964 

0.026 

30 

U.0U6 

0.990 

0.016 

31 

0  .004 

0.994 

0.010 

32 

0.002 

0.997 

0.006 

33 

0.001 

0.998 

0.003 

34 

0.001 

0.999 

0.002 

35 

0.000 

0.999 

0.001 

36 

0.000 

1.000 

0.001 

U3 

20.00 

X 

P(X  ) 

C(X  ) 

D  ( X ) 

7 

0.001 

0.001 

1.000 

8 

0 . 0U1 

O.U02 

0.999 

9 

0.003 

0.005 

0.998 

10 

0.006 

0.011 

0.99!) 

11 

0.011 

0.021 

0 , 989 

12 

0  »  018 

0.039 

0.979 

1  3 

0.027 

0.066 

0.961 

14 

0 . 039 

0.105 

0.934 

15 

0.052 

0.157 

0.895 

16 

0  •  065 

0.221 

0.843 

17 

0.076 

0.297 

U  ,  /  /  9 

18 

0.084 

0.381 

0  .  /U3 

19 

0.089 

U  .470 

0.619 

2  0 

0.089 

0.559 

0.530 

21 

0 . 085 

0.644 

0.441 

22 

0.077 

0 .7  21 

0.  356 
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X 

PtX) 

C(X) 

OCX) 

23 

0.067 

0.787 

0.279 

24 

0.056 

0.843 

0*213 

25 

0.045 

0.868 

0.157 

26 

0.034 

0.922 

0.112 

27 

0.025 

0.946 

0.078 

28 

0.018 

0.966 

0.052 

29 

0.013 

0.978 

0.034 

30 

0.006 

0.987 

0*022 

31 

0*005 

0*992 

0*013 

32 

0*003 

0  *995 

0*008 

33 

0*002 

0*997 

0*005 

34 

0*001 

0*999 

0*003 

35 

0*001 

0*999 

0*001 

36 

0*000 

1.000 

0*001 

U*  20.50 


X 

PtX) 

cm 

6 

0*001 

0*002 

9 

0*002 

0*004 

10 

0.005 

0*008 

11 

0.008 

0.017 

12 

0.014 

0.031 

13 

0*023 

0*054 

14 

0*033 

0.087 

15 

0*045 

0.132 

16 

0.056 

0.190 

17 

0*070 

0*261 

16 

0*080 

0*340 

19 

0*086 

0,426 

20 

0*086 

0.515 

21 

0*086 

0*601 

22 

0*080 

0*661 

23 

0*072 

0*753 

24 

0.061 

0.814 

25 

0*050 

0.864 

26 

0*040 

0.904 

27 

0*030 

0*934 

26 

0*022 

0.956 

29 

0*016 

0*971 

30 

0*011 

0*982 

31 

0.007 

0.989 

32 

0.  004 

0 .993 

P<X) 

0.999 

0.998 

0.996 

0.992 

0.983 

0.969 

0.946 

0.913 

0.888 

0.810 
0.739 
0 . 660 
0.574 
0.485 
0.399 
0.319 
0.247 
0.186 
0.136 
0.096 
0.066 
0.044 
0.029 
0.018 
0.  011 
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X 

P  l  X  } 

C  ( X ) 

p(  X  ) 

3  J 

6.003 

0.996 

0.007 

34 

0.002 

0.998 

0.004 

35 

0.001 

0.999 

0.002 

36 

0.001 

0.999 

0*001 

37 

0.000 

1.  000 

0.001 

L  = 

21.00 

X 

P<X) 

C<X  ) 

D(X) 

8 

0.001 

0.001 

1.000 

9 

0.002 

0.003 

0.999 

10 

0.003 

0  t  0 06 

0.997 

11 

0 .007 

0.013 

0.994 

12 

6.012 

0.025 

0.987 

13 

0.019 

0.043 

0.975 

14 

0.028 

0.072 

0.957 

15 

0.040 

0.111 

0.928 

16 

0.052 

0.163 

0.889 

17 

0.064 

0.227 

0.837 

18 

0.075 

0.302 

0.773 

19 

0.083 

0.384 

0.698 

20 

0.087 

0.471 

0.616 

21 

0.087 

0.558 

0.529 

22 

0.083 

0.640 

0.442 

23 

0.076 

0.716 

0.360 

24 

0 . 066 

0.782 

0.284 

25 

0.056 

0.838 

0.218 

26 

0.049 

0.883 

0.162 

27 

0.035 

0.917 

0.117 

28 

0 .026 

0.944 

0 .063 

29 

0.019 

0.963 

0.056 

30 

0.013 

0.976 
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MONITORING  OF  CONTRACTORS 
RELIABILITY  AND  MAINTAINABILITY  PROGRAMS 


FOREWORD 


Tne  purpose  of  this  section  is  to  provide  guidance  to  the  System  Program 
Offices  (3P0)  Reliability  and  Maintainability  (R/m)  Monitors  in  organization 
ana  operation  of  an  R/M  program  on  their  contracts. 

7. .«  duties  of  tne  Monitor  are  outlined  in  AF3CR  80-1  and  ESDR  80-2.  However, 
specific  instructions,  or  an  operating  manual  are  not  presently  available 
i  r  ar.v  other  AF  publication.  This  section  partially  fills  the  need  by  out-, 
w'ninq  steps  to  be  taken,  and  points  to  be  checked  as  the  work  progresses, 
tf.en  used  in  conjunction  with  USAF  Specification  Bulletin  506  it  considers 
R/V.  t.  rougnout  the  AF  cycle,  from  definition  through  acquisition  phases. 

•  evement  of  ESD  reliability  objectives  requires  that  the  Specification  and 
S* a'ement  of  Work  (SOW)  be  explicit  in  what  a  contractor  is  required  to  do,  - 
but'no  amount  of  legal  phraseology  can  produce  the  results  of  a  good  reliabiji  y 
engineering  program,  actively  pursued. 


SECTION  VII 

CONTENTS 


SUBJECT  PAGE 

1 .  Introduction  .  1 

a.  Explanation  of  Terms  .  1 

b .  Impact  of  R/M  on  Overall  Program  ...  1 

c .  R/M  Requirements  .  1 

d .  Approach  .  2 

e.  R/M  Functions  .  2 

f.  R/M  Monitor’s  Function  .  3 

2.  Preparation  for  Monitoring  .  4 

a.  Precontract  .  4 

b.  First  Post  Contract  Award  Meeting  ..  5 

c .  Second  Post  Contract  Award 

Meeting  . 6 

d.  Elements  of  the  Reliability 

Program  Flan  .  8 

3.  Monitoring  Procedures  . . 9 

a.  Weak  Link  Procedures  . *  9 

b.  Failure  Reporting  and  Corrective 

Action  Loops  . 9 

c.  Engineering  Change  Proposals  (ECPs) 

Hon-ECPs,  and  Contract 
Modifications  .  .  11 

d.  Test  and  Demonstration  Plans  .  12 

e.  Agreement  on  Terms  and  Definitions  .  13 

f.  Critical  Parts  .  13 

g.  Control  of  Subcontractors  .  l4 

h.  Manuals  for  Training,  Installation, 

and  Maintenance  .  14 

1 .  In-House  Manufacturing,  Packaging, 

Storing,  and  Transporting 
Procedures  .  15 

j .  Spares  and  Aerospace  Ground 

Equipment  (AGE)  . 15 

k.  Indoctrination  and  Training  . .  l6 

l.  R/M  Math  Model  . * . . .  l£> 


SUBJECT 


PAGE 


L.  General . . . •  ••••  17 

a.  Use  of  Reports  . . 17 

b.  Records  . ••••••.••  17 

c.  Plant  Visits  . 1? 

d.  Problem  Resolution  . 20 

e.  Assistance  Available  to 

the  Monitors  . .  20 

ILLUSTRATIONS 

FIGURE 

I.  Example  of  Reliability  Math  Model  ....  23 

II.  Typical  Reliability  Growth  Cycle  .  25 

III.  Typical  Data  Flow  Diagram  For 

Reliability  Organization  .  27 

IV.  Example  of  Failure  Report  Form  .  29 

Vm  Example  of  Operating  Time  Log  .  31 

VI.  Maintenance  Discrepancy/Production 

Credit  Record  .  33 

VI I.  Example  of  Modification  Analysis 

Sx-mary  Form  .  35 

VIII.  Example  cf  Corrective  Action 

Fcllcv-Up  Report  . 37 

APPENDIX 

!•  Development  of  Circuit/System 

Mathematical  Model  .  39 


iv 


MONITORING  OF  CONTRACTORS 
RELIABILITY  AND  MAINTAINABILITY 
PROGRAMS 


1.  Introduction : 

a.  Explanation  of  Terms : 1 

(1)  Reliability.  The  probability  that  a  given  system/equipment 
will  perform  its  intended  function,  without  failure,  for  a  specified  period 
of  time,  when  operated  in  its  prescribed  manner. 

(2)  Maintainability  (Operational).  The  probability  that  when  a 
maintenance  action  is  initiated  under  stated  conditions  a  failed  system/ 
equipment  will  be  restored  to  operable  condition  within  a  specified  total 
downtime . 


b.  Impact  ol  R/M  on  Overall  Program.  The  reliability  and  maintainability 
inherent  in  any  given  system  affects  the  overall  system  operation.  These 
effects  include  the  quantity  of  spares  which  must  be  provisioned  to  support 
it,  the  amount  of  maintenance  which  will  be  required  to  achieve  a  given 
degree  of  operational  readiness,  and  the  number  of  systems  required  to  do 

a  particular  job,  i.e.,  target  coverage,  continuous  satellite  relay  capa¬ 
bility,  continuous  operation  for  a  given  period,  etc .  These  R/M  parameters 
also  determine  the  number  of  systems  required  to  accomplish  a  given  set  of 
objectives,  the  magnitude  of  development  funds  required,  the  skills  and 
facilities  required  for  development  and  production,  and  the  amount  of  time 
required  to  evolve  from  the  definition  to  the  operational  phase. 

c.  R/M  Requirements.  For  a  given  system  or  equipment,  requirements 
are  based  upon  the  Specific  Operational  Requirements  (SOR).  The  SORs  are 
established  by  USAF  after  study  of  AF  defense  requirements.  Usually  trade¬ 
offs  are  necessary  to  fit  the  requirements  within  budget  or  time  limitations . 
These  may  involve  Reliability  versus  Maintainability  to  achieve  a  certain 
overall  Availability  (A) ;  Reliability  versus  Cost;  Reliability  versus  Sched¬ 
ule;  Reliability  versus  Performance,  etc.  Once  the  R/M  requirements  have 
been  established  it  is  imperative  that  they  be  achieved  since  they  have  a 
direct  impact  on  every  major  element  of  the  program. 


J-For  additional  terms  pertaining  to  R/M  matters  see  MIL- STD- 721  and 
MIL- STD- 829. 
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d.  Abroach ■  fh«.  requirements  will  be  achieved  only  by  good 
planning  and  vigorous  management.  To  achieve  these  objectives  the  SPO 
must  define  the  program  in  terms  of  detailed  R/M  requirements. 

To  monitor  progress  on  the  contract  and  ultimate  achievement  of  the 
requirements  necessitates  development  of  an  R/M  Program  Plan1  with  specific 
tasks  and  interim  objectives,  which  can  be  audited  on  a  time-phased  basis 
to  determine  accomplishments . 

e.  R/M  Functions*  The  list  of  functions  or  activities  which  should 
be  performed  during  system  development  and  acquisition  can  be  expanded 
almost  without  bound.  A  typical  list  of  key  functions  for  which  an  input 
on  reliability  and  maintainability  is  required,  and  which  may  involve 
direct  contributions  by  an  R/M  Monitor,  is  summarized  below.  By  expanding 
upon  this  basic  framework  a  Monitor  can  develop  a  full  R/M  program.  These 
functions  have  been  abbreviated  and  grouped  according  to  the  various  phases 
of  system  development. 

MAJOR  EVENT 


Conceptual  Fhase 
Feasibility  Study 


Publish  SOR 


Definition  Fhase 

Prepare  Work  Statements 
Select  source 
Initiate  work 


Conduct  trade-offs 


Establish  base  line 


RELIABILITY /MAINTAINABILITY  PARTICIPATION 


Identify  achievable  R/M.  Identify 
potential  problems.  Outline  approach 
required . 

Provide  R/M  quantitative  requirements 
or  objectives  for  inclusion  in  SOR. 


Frovidc  program  requirements  and  tasks . 

Evaluate  bidders  proposals . 

Brief  contractors  on  approach,  policy, 
etc . 

Provide  data  and  technical  direction/ 
evaluation . 

Provide  analysis  and  final  requirements. 


1 Program  plans  have  been  covered  in  ESDP  80-2. 
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major  Evmrr  reliability /mauttaikability  participation 

Define  program  Establish  program  elements,  applicable 

documents ,  and  approved  program  plans . 


Acquisition  Phase 

Let  contracts 


Provide  work  statements. 


Begin  design 

Finalize  design 
Conduct  testing# 

Evaluate  results 

Manufacture 

Shipping  and  storage 
Installation  and  checkout 

Operational  Fhaae 

Activate 


Provide  analysis.  Control  parts 
selection.  Approve  subcontractors. 
Design  tests . 

Provide  design  reviews .  Approve 
specifications,  drawings,  etc. 

Provide  test  requirements .  Design 
teats.  Interpret  data. 

Identify  discrepancies.  Establish 
corrective  action.  Retest  as  necessary. 

Coordinate  all  process  controls,  tests, 
and  inspections  with  Quality  Control. 

Evaluate  procedures  and  controls. 

Monitor  progress,  problems,  and 
corrective  actions .  Phase  operations 
data  into  basic  data  syatem. 


Transition  from  acquisition  data  to 
operational  data  for  me  asurement  and 
control  of  reliability  and  maintainability. 

Verify  any  updating  programs- 

Activate  and  support  in-service  engi¬ 
neering  functions . 


f .  R/M  Monitor's  Function: 

(l)  The  R/M  Monitor’s  primary  responsibility  ic  to  serve  as  the 
SPO  focal  point  for  R/M  activities.  The  complete  responsibilities  are 
clearly  delineated  in  ESEP  80-2  and  ESDR  80-U.  They  encompass  work  within 
the  SPO,  the  supporting  activities,  and  direct  monitoring  of  the  contractor 
and  subcontractors. 
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(2)  The  SPO  Monitor  does  not  replace  the  contractor’s  R/M  organi¬ 
zation  but  serves  to  supplement,  analyze,  and  control  the  contractor’s  work. 

The  need  for  the  AF  Monitor's  position  jis  based  on  the  fact  that  contractors 
are  of  necessity  profit  oriented  -while  the  AF  must  be  mission  oriented. 

(3)  The  Monitor  serves  a  vital  role  in  achievement  of  tbe  overall 

system/equipments  requirements.  While  AF,  AFSC,  and  ESD  regulations  define 
tbe  Monitor's  responsibilities,  AF  publications  do  not  cover  instructions 
on  development  and  operation  of  an  R/M  program.  While  this  pamphlet  is 
not  all  encompassing  it  does  cover  the  important  steps  in  operating  a 
program,  and  points  out  some  problem  areas  for  particular  attention  in 

monitoring.  These  items  are  based  on  experience  on  other  R/M  programs  and 

will  assist  the  monitors  in  getting  their  programs  into  operation  smoothly 
and  quickly. 

2.  Preparation  ior  Monitoring: 
a .  Precontract: 

(1)  The  Monitor  should  review  the  specifications  and  SOW,  and 

develop  a  monitoring  plan  based  on  tbe  tasks  described.  If  tbe  SPO  is 
specific  and  detailed  in  the  precontract  stage,  the  easier  will  be  the  work 

in  tbe  later  stages.  Detail  what  is  to  be  covered.  Note  that  some  items 

in  R/M  are  deliverable;  i.e.,  plans  or  reports,  others  are  requirements 
which  will  require  checking  in-plant.  Determine  which  items  may  require  a 
visit  to  the  contractor's  plant  and  which  items  can  be  reviewed  at  ESD^. 

The  important  point,  is  to  develop  a  check  list  of  what  is  expected  from 
the  contractor.  The  Monitor  should  be  prepared  at  the  Bidders'  Briefing 

to  outline  what  is  required. 

(2)  One  of  the  basic  monitoring  problems,  assuming  that  the  con¬ 
tract  and  specification  are  adequate,  is  that  contracts  and  specifications, 
with  few  exceptions,  establish  a  requirement  which  is  based  on  end- item 
test  (e.g.,  1000  hours  mean  time  between  failures  (MTBF)  or  99.99 $  avail¬ 
ability)  of  the  final  product  but  do  not  provide  any  interim  check  points. 

This  means  that  tbe  contractor  can  start  design  and  continue  on  through 
production,  to  final  test  before  the  SPO  is  aware  that  R/M  requirements  will 
not  be  met.  This  may  result  in  a  situation  where  delivery  will  become  of 
paramount  importance  and  the  Government  is  forced  to  accept  substandard  items 
or  face  a  long  delay  to  reprocure  the  equipment.  Most  contracts  allow  progress 


1A  new  procedure  covering  the  use  of  DOD  Form  1423,  Contractor  Data 
Requirements  List,  is  being  prepared  at  ESD.  It  will  be  incorporated 
into  tbe  contracts  and  spell  out  all  deliverable  data  type  items. 
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payments.  If  these  payments  are  controlled  in  an  adequate  manner,  tbe 
Monitor  can  do  much  to  assure  attainment  of  AF  goals.  One  of  the  legal 
means  for  controlling  this  is  to  set  intermediate  requirements  and  monitor 
the  work  as  it  progresses. 

(3)  It  should  be  noted  that  there  are  two  general  types  of 
specifications : 

(a)  A  "performance  type"  specification  in  which  the  con¬ 
tractor  is  given  the  problem  and  the  end  result  desired.  Within  certain 
limitations,  standard  parts,  etc.,  the  contractor  works  toward  the  goal. 

(b)  An  "equipment  type"  specification,  in  which  the  con¬ 
tractor  is  told  what  to  use,  "off-the-shelf"  Government  Furnished  Equip¬ 
ment  (GFE),  etc.,  and  there  is  less  chance  for  "a  state-of-the-art"  break¬ 
through  . 


Request  for  Proposal  (RFF)  statements  to  the  bidders  must  be  specific 
as  to  what  is  expected  of  them.  The  work  involved  on  these  types  of  con¬ 
tracts  differs.  When  the  contractor  is  developing  new  designs  and  approaches, 
the  emphasis  is  placed  in  the  design  stage.  Where  the  contractor  is  directed 
to  use  existing  equipments  the  emphasis  shifts  to  determining  the  equip¬ 
ments  to  be  used,  what  weak  points  have  been  previously  noted,  and  how  tbe 
system  should  be  configured. 

(4)  Initially,  the  Monitor  should  set  up  his  own  listings  of  major 
programs  (systems  and  subsystems) .  Identify  who  in  the  SPO  is  assigned 
responsibility  as  Project  Engineer,  Production  Specialist,  Buyer,  RADC 
Support,  RADC  Engineer,  etc.  In  each  case  list  the  telephone  number,  posi¬ 
tion  title,  and  organization.  A  list  of  specifications  applicable,  R/M 
intermediate  and  final  goals,  SPO  estimates  of  time-phasing,  manloading, 
and  task  duration,  should  also  be  estimated. 

b .  First  Post  Contract  Award  Meeting: 

(1)  At  this  point,  after  award  of  contract,  the  Monitor  can  give 

a  more  specific  briefing  to  contractor.  The  Air  Force  Procurement  Instruc¬ 
tions  (AFPIs)  require,  on  larger  contracts,  that  these  briefings  be  held 
within  30  days  of  contract  award.  If  not  scheduled  by  the  contracting  officer 
the  Monitor  should  request  that  the  contractor's  R/M  group  meet  with  the  SPO 
R/M  representatives.  At  this  meeting  the  Monitor  can  give  direction  and 
clarification  of  specific  tasks  if  required. 

(2)  Request  the  contractors  R/M  Plan  be  submitted  as  early  as 
possible  so  it  can  be  reviewed.  Determine  if  contractor  understands  all 
R/M  aspects  required  in  tbe  contract,  and  if  his  general  approach  and  under¬ 
standing  follows  the  Monitor's  planned  check  list. 
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(3)  It  is  possible  that  some  problems  may  arise  at  the  ’nee t lag 
that  cannot  be  resolved  at  that  time.  The  problem  should  be  discussed 
and  defined.  Then  specific  assignment  should  be  made  to  a  person  or 
activity;  e.g.,  contractors  design  group  or  SPO  contracting  office,  to 
follow  the  matter  through*  In  addition  to  assigning  the  problem  for 
resolution,  a  date  for  reporting  on  the  progress  or  resolution  should  be 
established.  These  problems  and  assignments  should  be  incorporated  into 
the  minutes  of  meeting  as  a  method  of  checking  progress . 

(4)  Establish  the  working  arrangements  and  contact  to  be  used 
in  development  and  operation  of  the  program.  The  contractor  should  be 
given  an  organization  chart  of  the  SPO,  with  personnel  assigned  to  various 
functions,  and  he  should  be  requested  to  provide  similar  information  to 
the  SFO. 


(5)  The  Monitor  should  establish  certain  policies  with  respect  to 
all  meetings  with  the  contractor.  As  a  minimum  these  should  include; 

(a)  That  an  agenda  be  prepared  and  distributed  in  advance  of 

each  meeting. 

(b)  That  minutes  of  each  meeting  be  kept,  and  be  distributed 
to  all  parties  concerned. 

(c)  That  problems  pending  or  resolved  be  included  in  the  min¬ 
utes  (see  (3)  above). 

(d)  That  attendance  at  the  meetings  be  kept  as  small  as 
possible,  consistent  with  the  work  to  be  covered. 

(e)  That  regular  meetings  be  held  with  contractor  personnel 

to  check  progress  made.  A  frequency  of  about  four  to  six  week  intervals  has 
been  found  to  be  practical.  The  meeting  dates  should  be  adjusted  as  needed 
to  check  contract  mile  posts. 

c .  Second  Fcst  Contract  Award  Meeting: 

(1)  Review  contractor's  R/M  Plan  in  detail  with  his  representatives. 
The  discussion  should  cover  the  contractors  plan  on  a  task  by  task  basis 

and  assure  that  all  contractual  requirements  have  been  met.  If  any  points 
have  not  been  covered  in  the  plan,  or  if  the  statement  of  the  work  to  be 
done  is  not  clear,  the  plan  or  the  contract  should  be  amended.  If  this 
involves  changes  in  price,  delivery,  etc.,  written  approval  of  the  con¬ 
tracting  officer  must  be  obtained. 

(2)  Attention  should  be  paid  to  the  time-phasing,  manloading,  and 
task  duration  elements  of  the  plan.  Comparison  of  the  contractor's  alloca¬ 
tions  should  be  made  against  the  SPO  Monitor's  estimates.  (ROTE:  By  estab¬ 
lishing  these  items  early  in  the  contract,  logical  redirection  of  the  pro¬ 
gram  can  be  made  later  without  incurring  major  overruns.) 
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(3)  Obtain  from  the  contractor  a  list  of  his  major  subcontractors, 
their  schedules,  the  guidance  documents  used  to  control  them,  the  reporting 
procedures  used  for  in-house,  and  for  external  control.  These  documents 
should  be  approved  by  the  SPO  prior  to  implementation.  Final  approval  of 
the  contractor  R/M  Flan  should  be  contingent  on  approval  of  these  items. 

(4)  Agreement  should  be  obtained  on  reporting  periods,  report 
format,  and  content.  A  simple  but  well  documented  engineering  letter  type 
report  is  desired.  The  emphasis  should  be  on  report  content,  not  appearance 
Depending  on  the  program  involved,  scope  and  duration,  it  may  be  acceptable 
to  have  the  R/M  reports  as  part  of  the  overall  progress  report  rather  than 
as  separate  reports. 

(5)  Get  the  R/M  Math  Model  usually  required  in  the  program  plan. 
This  will  serve  as  the  base  for  checking  progress  on  the  contract.  It 
should  be  updated  and  revised  as  the  design  and  work  progresses.  (See 
Figure  I.) 

(6)  Set  up  the  specific  monitoring  points.  (See  Figure  II, 

"Typical  Growth  Cycle”.)  Very  few  ESD  contracts  will  fit  this  completely, 
but  some  points  can  be  combined  and  the  general  trend  should  resemble  this 
chart.  Note  particularly  the  curves  and  the  Reliability  values  shown: 

(a)  Specified  Reliability.  This  is  the  contractual  require¬ 
ment,  shown  as  a  constant  value.  This  value  is  required  as  the  end  result 
on  completion  of  the  contract.  In  ESD  contracts  it  represents  the  "oper¬ 
ational"  reliability  required  of  the  system/equipment . 

(b)  Predicted  Reliability.  This  curve  represents  the  con¬ 
tractor's  predicted  approach  on  a  phased  basis,  to  achievement  of  the 
specified  reliability.  In  some  cases,  in  the  early  stages  of  development, 
the  predicted  values  may  be  below  requirements.  IXxring  this  period  the 
contractor  should  be  expending  engineering  effort  to  improve  the  design  to 
assure  attainment  of  the  goals.  It  should  be  noted  this  curve  must  ulti¬ 
mately  exceed  the  requirements  in  order  to  assure  achievement  of  the 
specified  values.  This  is  to  allow  for  the  inevitable  degradation  of  the 
inherent  reliability  due  to  production,  handling,  etc .  These  interim 
predicted  values  should  be  obtained  from  the  contractor  and  will  allow  for 
assessment  of  progress . 

(c)  Reliability  Status.  This  curve  represents  the  achieved 
results  based  on  test  or  demonstration  at  the  agreed  on  monitoring  points. 
The  results  should  equal  or  exceed  the  predicted  values. 

NOTE:  These  curves  are  the  basic  source  of  information  for  comple¬ 
tion  of  the  Reliability  Status  Report  required  by  AFSCR  80-1, (RCS:  AFSC-R44) 
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Hot  a3 i  contracts  will  follow  these  steps.  In  some  cases  equipment 
is  GFE,  "off-the-sbelf”,  modified  GFE,  etc.,  but  this  cbart  can  be  modified 
as  needed  to  allow  for  these  variations  and  still  give  intermediate  evalu¬ 
ation  points.  These  Mile  Posts  should  be  used  to  REVIEW,  MEASURE,  REPORT, 
and  CORRECT.  Whenever  the  results  do  not  appear  to  follow  the  predicted 
curves  the  monitor  should  take  action.  Do  not  wait  until  tbe  contract  has 
run  so  long  that  the  SPO  cannot  consider  termination  or  alternate  approaches 
because  of  lack  of  time. 

d.  Elements  of  the  Reliability  Program  Flan.  Depending  on  the  SOW 
tasks  and  the  specifications  referenced,  tbe  contractor  is  required  to  per¬ 
form  many  of  the  following  functions  in  his  R/M  program.  The  list  is  not 
complete  and  the  depth  that  each  item  will  have  is  dependent  on  tbe  SOW. 

(1)  Develop  R/M  Plan. 

(2)  Develop  and  update  tbe  matb  model. 

(3)  Conduct  critical  item  studies. 

(U)  Conduct  special  studies. 

(5)  Develop  and  implement  a  plan  for  control  of  subcontractors. 

(6)  Perform  human  factors  analysis. 

(7)  Develop  manufacturing  and  handling  control  procedures. 

(8)  Plan  for  and  conduct  training  and  indoctrination  of  personnel. 

(9)  Establish  and  operate  a  closed  loop  data  collection  and  analysis 

system. 

(10)  Plan  for  and  conduct  design  reviews. 

(11)  Analyze  and  evaluate  modifications  and  Engineering  Change 
Proposals  (ECPfc) . 

(12)  Provide  for  a  corrective  action  system  for  all  phases  of  con¬ 
tract  including  sites,  if  applicable. 

(13)  Develop  a  test  plan. 

(lU)  Provide  for  test  and  evaluation. 

(15)  Establish  reporting  procedures. 


8 


(16)  Provide  planning  for  AGE  and  spares  considerations. 

(17)  Check  manuals;  operating,  installation,  and  repair. 

The  R/M  Monitor's  function  is  to  determine  that  the  plan  is  adequate 
and  is  followed  completely. 

3 .  Monitoring  Procedures : 

a.  Weak  Link  Procedures .  Weak  link  studies  may  be  one  of  the  items 
required  by  the  SOW  under  special  studies.  In  developing  the  specific  tasks 
for  the  SOW  consideration  should  be  given  to  requiring  the  contractor  to  list, 
in  his  regular  or  special  reports,  those  "weak  links”  or  limiting  factor 
items  that  have  been  found  as  work  on  the  contract  progresses .  These  are 

a  distinct  consideration  from  those  items  that  would  prevent  tbe  contractor 
from  meeting  bis  contractual  requirements.  Where  a  contractor  fails  to 
meet  tbe  specified  contractual  requirements  corrective  action  is  required 
by  the  contractor.  In  the  case  of  a  "weak  links”  requirement  tbe  contractor 
will  deliver  tbe  required  reliability  or  maintainability  but  must  point  out 
the  limiting  factors  found,  and  detail  what  could  be  obtained  with  a  given 
amount  of  engineering  effort  and  production  time  to  improve  the  system/ 
equipment.  The  monitor  should  study  and  evaluate  these  weak  links  and 
recommend  a  course  of  action  to  tbe  SPO . 

b .  Failure  Reporting  and  Corrective  Action  Loops:1 

(1)  The  contractor  should  detail  bis  failure  reporting  and  correc¬ 
tive  action  loop  as  part  of  bis  overall  R/M  Plan.  In  many  cases,  these 

may  be  part  of  his  Quality  Control  manual  which  is  required  under  MIL-Q-9858. 
The  essential  ingredients  of  any  system  are: 

(a)  All  failures  or  problems  are  reported  and  fully  documented. 
The  type  of  data  and  details  may  vary  depending  on  tbe  phase  of  the  contract. 

(b)  Analysis  and  tabulation  of  the  data. 

(c)  Actions  taken  based  on  the  data  available. 

(d)  Evaluation  of  the  actions  taken. 

(2)  Typical  data  flow  and  feedback  loops  in  various  parts  of  the 
system  cycle  are  shown  as  Figure  III. 


*For  additional  information  on  R/M  Data  Collection  and  Evaluation  System 
s  ee  ESDP  80- 3 • 
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(3)  Various  forms  are  used  in  collection  of  data.  Each  failure 
reported  should  contain  sufficient  information  to  enable  the  contractor  and 
SPO  to  make  a  proper  evaluation  of  the  trouble.  Tfce  essential  information 
elements  for  R^M  purposes  are  listed  in  MIL-R-275^2  and  MXL-M-26512.  To 
get  all  these  elements  may  require  two  or  more  forms.  IXie  to  the  nature 
of  E3D  systems,  where  redundancy  is  common,  provision  must  also  be  made 
to  get  tbe  operating  time  of  the  equipment/s ys terns .  Since  equipments  are 
not  always  operating  due  to  redundancy,  Preventative  Maintenance  (PM)  or 
Corrective  Maintenance  (CM)  provision  must  be  made  to  obtain  true  operating 
time  as  the  work  progresses.  A  typical  form  is  shown  in  Figure  IV  which  is 
jsed  to  record  an  individual  failure,  and  Figure  V  which  is  an  operating 
time  log  used  to  determine  operating  time  on  a  given  drawer  or  cabinet. 

(U)  Points  to  watch  in  data  collection: 

(a)  By  personal  observation  and  contacts  with  contractor 
personnel  determine  if  all  failures  are  recorded.  Log  books  can  be  checked 
against  failure  reports  to  determine  if  the  information  bas  been  documented. 
Bv  checking  supply  room  issue  or  requisition  slips  an  indication  of  equip¬ 
ment  failures  and  parts  replacement  can  be  obtained  for  cross  checking 
purposes. 


(b)  Quality  of  entries  is  very  important.  For  simplicity  and 
ease  of  operation,  most  contractor  data  systems  use  coded  entries.  In 
checking  entries  tbe  accuracy  of  coding  is  important.  Equally  important, 
however,  is  tbe  determination  of  whether  the  failure  is  a  primary  or 
secondary  failure,  whether  it  was  an  engineering  design  problem  or  a  produc¬ 
tion  problem,  etc.  Contractual  provision  should  be  made,  both  on  site  and 
in  the  factory  for  the  assistance  of  a  qualified  R/M  engineer  to  work  with 
operating  and  maintenance  personnel  in  evaluation  of  the  causes  of  failure. 

(c)  Cross  checking  of  clock  time  versus  elapsed  time  meters. 
Generally,  command  and  control  systems  have  one  or  more  Elapsed  Time 
Indicators  (ETIs)  per  subsystem.  For  proper  analysis,  operating  time  must 
be  kept  on  a  discrete  basis,  such  as  a  drawer  or  cabinet  level.  Periodic 
checks  of  clock  time  versus  ETI  time  will  give  an  indication  of  the  accuracy 
of  the  records. 

(d)  Parts  analyses  records.  Simple  recording  of  parts 
failures,  and  replacement  actions  does  not  suffice  for  complete  R^M  work. 
Causes  of  repetitive  failures  or  those  which  are  critical  to  system  oper¬ 
ation,  must  be  determined.  To  accomplish  this  may  require  a  laboratory 
analysis  of  the  failure.  Figure  VI  is  a  sample  form  for  use  in  following 
these  items  through  tbe  repair  and  return  cycle.  Rote  the  detailed  infor¬ 
mation  required  in  completion  of  these  forms.  It  is  imperative  that  this 
information  be  complete  and  accurate  if  adequate  performance  is  to  be 
obtained. 
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(e)  The  fact  that  a  record  is  made  of  tbe  parts  failure  at 
the  test  site  or  proving  ground,  and  that  a  copy  of  the  form  was  sent  to 
tbe  prime  or  integration  contractor  is  not  proof  that  corrective  action 
has  been  taken.  When  visiting  the  plant  the  monitor  should  check  to  deter¬ 
mine  the  action  taken  and  results  obtained.  This  should  be  done  by 

starting  with  the  first  office  receiving  the  report  and  then  following 
through,  office  by  office,  through  recording,  analysis,  engineering, 
production,  test,  etc.  Particular  attention  should  be  paid  to  those 
problems  when  failures  are  critical  or  repetitive. 

(5)  Use  of  AFM  66-1,  Maintenance  System  Reporting.  During  FT  6h 
tbe  AF  will  institute  a  Maintenance  Data  Collection  System  (MDCS)  which  is 
similar  to  the  system  outlined  in  AFM  66-1,  presently  the  principle  AF 
system  for  maintenance  and  logistic  system  reporting.  A  complete  descrip¬ 
tion  of  the  system  and  instructions  for  its  use  will  be  contained  in  the  re¬ 
vision  of  AFSCM  375*1  soon  to  be  published.  It  is  expected  that  this  MDCS 

is  to  be  instituted  not  later  than  tbe  start  of  Category  II  testing  (REF: 

AFR  80- 14) .  To  assure  proper  implementation  of  the  MDCS,  arrangements  must 
be  made  for  Work  Unit  Coding  (WUC)  early  in  the  contract.  Assignment  of 
WUC  is  the  responsibility  of  ROAMA. 

c •  Engineering  Change  Proposals  (ECPs),  Non-ECPs,  and  Contract  Modifi¬ 
cations  .  In  most  contracts  for  large  equipments  or  systems  which  axe  push¬ 
ing  the  "state-of-the-art”,  changes  frequently  become  necessary.  Depending 
on  tbe  type  of  change.,  and  the  time  phasing,  these  may  be  ECP,  non-ECP,  etc. 
The  definitions  of  these  terms  varies  from  contractor  to  contractor.  The 
contractor  should  be  required  to  analyze  proposed  changes  for  effect  on  R/M. 
The  change  request  should  state  clearly: 

(1)  What  is  involved,  the  part,  component,  or  equipment  affected. 

(2)  Why  change  is  proposed  and  reason  for  request. 

(3)  Number  and  location  of  equipments  or  systems  involved. 

(*0  Cost  per  unit  and  total  cost  to  effect  the  change. 

(5)  Effect  on  production/delivery  schedules. 

(6)  Affect  on  R/M  of  the  system/equipment ♦ 

(7)  How  the  effect  of  the  proposed  change  will  be  verified. 

All  statements  should  be  clear  and  quantified;  e.g.,  the  MTBF  is  presently 
100  hours.  By  changing  the  cut-out  switch  in  the  power  supply  to  type  XT 
which  has  a  lower  Failure  Rate,  150  hours  KTBF  will  be  obtained.  Example 
for  form  to  used  in  reporting  these  proposed  changes  is  shown  as  Figure 
VII  and  Figure  VII-A.  If  the  base  line  configuration  has  been  established,  the 
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R/M  Monitor  must  assure  that  these  proposed  changes  are  processed  through 
the  SPO  Configuration  Control  Board  and  that  the  R/M  activity  has  a  voice 
in  the  decision  of  the  hoard. 

d.  Test  and  Demons tratl on  Plana,  General  considerations  of  the  test 
programs  in  Category  testing  are  covered  In  AFR  80-lU  and  are  further 
amplified  in  ESDP  375-2,  A  Typical  Test  Plan  for  Electronic  Systems,  a 
pamphlet  vhich  has  been  published  by  ESTT. 

The  R/M  specifications  require  the  contractor  to  develop  test  plans 
to  demonstrate  achievement  of  the  required  R/M  characteristics  to  a  given 
confidence  level.  It  should  be  noted  that  Hq  AF5C  has  directed  that  R/M 
shall  be  specified  in  all  system  contracts  and  that  attainment  shall  be 
demonstrated  prior  to  final  payment. 

Points  for  special  attention  in  Test  Programs: 

(l)  In  establishing  requirements  consideration  must  be  given 
to  the  degree  of  confidence  and/or  risk  required  since  the  cost  of  testing 
increases  very  sharply^  when  confidence  limits  in  excess  of  9°^  are  speci¬ 
fied. 


(2)  In  complex  redundant  systems  demonstration  of  overall 
system  reliability  may  require  lengthy  test  times.  Careful  analysis  may 
allow  for  demonstrations  on  a  subsystem  or  component  basis  that  could  be 
accepted.  Careful  check  of  system  interface  and  switching  needs  should  be 
made  in  determining  the  test  approach. 

(3)  Contractor/SFO  agreement  on  approach  for  accumulation  of 
test  time  and  repair  time  should  be  established.  Ideally,  the  tests  for 
R/M  should  be  run  independently  of  performance  and  other  tests .  Schedules 
frequently  do  not  allow  sufficient  time  for  this.  R/M  Monitors  should  re¬ 
quest  ample  time  for  running  of  R/M  tests  in  accordance  with  applicable 
specifications,  but  all  data  accumulated  during  other  tests  should  be  used 
if  possible.  The  data  used  must  be  properly  evaluated.  For  example: 

(a)  If  checking  devices  such  as  diagnostic  routines  or 
performance  monitors  are  not  available  the  maintenance  time  may  be  higher 
than  normal . 


(b)  Checks  must  be  made  to  determine  that  equipment  is 
actually  performing  its  function  and  not  simply  that  the  ETI  is  running 


^For  additional  information  on  establishing  test  time  requirements  and  risks 
see  ESDP  8O-5. 
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or  that  the  power  switch  is  "OK”.  Failures  do  occur  which  are  not  detected 
and  this  may  give  erroneous  reliability  values.  Definite  agreement  must 
be  reached,  prior  to  start  of  test,  on  how  the  system  is  to  be  "exercised” 
and  checked  out. 


(U)  Due  to  the  "concurrency  concept"  applicable  on  many  ESD 
programs,  shipment  and  installation  of  equipment  sometimes  occurs  before 
testing  or  retrofit  has  been  accomplished.  Careful  check  should  be  made 
as  to  assignment  of  AF  responsibilities  and  procedural  details  on  the 
following : 


(a)  Conditional  acceptance  at  plant. 

(b)  Engineering  responsibility  for  data  evaluation  and/or 

design. 

(c)  Site  engineering  and  installation  acceptance. 

(d)  Retrofit  acceptance  and  checkout. 

(e)  Final  test  and  acceptance. 


(5)  Environmental  test  conditions  should  be  specified  in  the 
test  plan  and  should  on  in  conformance  with  the  original  equipment/system 
specifications.  On  a  system  basis  the  environment  control  is  sometimes 
limited.  In  equipment  test,  adequately  controlled  test  chambers  can  usually 
be  obtained.  Care  should  be  taken  in  specifying  the  conditions  and  in 
checking  the  controls.  When  original  specifications  do  not  cover  this  point 
agreement  should  be  reached  prior  to  start  of  the  test.  Consideration, 
should  be  given  to  use  of  Advisory  Group  on  Reliability  of  Electronic 
Equipment  (AG  FE)  test  conditions. 


e.  Agreement  or  Terms  and  Definitions.  Prior  to  start  of  test,  pref¬ 
erably  as  part  of  the  basic  r/m  Plan,  agreement  should  be  reached  on 
definition  of  ^crns,  including  modes  of  operation,  equipment  and  systems 
failures,  up-' ire  ratios,  mission  success,  etc.  Basic  references  for  these 
terms  are  fou  F.IL-mTP-721,  MIL-STD-829  and  glossaries  given  in  MIL- 

R-275^2  and  KIi.-»  -  512.  Special  consideration  must  be  given  to  the  degree 

of  control  th*-  contractor  has  over  the  equipment  and  subsystems  involved. 

If  the  contractor  is  r.ot  responsible  for  the  intersite  communications  or 
for  certain  GFE  used,  the  definitions  must  be  adjusted  to  cover  these  items. 


f .  Critical  Parts: 

(l)  Some  Items  require  special  attention  in  program  monitoring. 
These  include : 


(a)  Items  for  which  adequate  R/M  data  is  not  available. 
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(b)  Items  haring  limited  life  due  to  natural  ageing  factors 
or  degradation  during  test . 


(c)  Items  requiring  special  handling  during  production, 
storage,  or  Issue. 

(2)  All  of  the  above  should  be  specially  documented  and  necessary 
planning  and  action  taken  to  avoid  system  degradation.  These  plans  and 
actions  should  be  checked  by  the  Monitor. 

g.  Control  ol  Subcontractors.  Standard  Government  procedure  and  civil 
contract  lav  establishes  relationship  only  between  the  customer  (SFO)  and 
the  prime  contractor.  Contract  stipulation  Is  usually  made  that  the  prime 
contractor  Is  responsible  for  his  subcontractor^  work,  requiring  that 
actions  involving  subcontractors  should  be  directed  through  the  prime  con¬ 
tractor.  Direct  Government/ subcontractor  negotiations  would  require 
specific  written  authorization. 

Prime  contractors  are  expected  to  prepare  Instructions  or  technical 
operation  procedures  which  supplement  their  normal  purchase  orders  and  define 
the  R/M  programs  required  of  their  subcontractors.  The  purchase  order  should 
state  the  R/m  requirements  levied  on  the  subcontractor.  These  requirements 
should  be  consistent  with  the  overall  requirements  of  the  system/equipment 
contract  and  should  normally  call  for  demonstration  of  achievement.  Prior 
to  approval  by  the  SFO  of  the  prime  contractors  R/M  program  or  test  plans 
the  subcontractors'  plans  should  be  checked  to  assure  that  the  entire  pro¬ 
gram  Is  In  conformance  with  the  system/equipment  contract. 

One  critical  point  to  be  checked  is  the  test  and  acceptance  point 
of  the  subcontractors'  output.  Another  critical  point  is  defining  responsi¬ 
bility  for  solution  of  interface  problems.  These  two  points  should  be 
covered  clearly  In  the  subcontractor  control  procedures. 

h.  Manuals  for  Training,  Installation,  and  Maintenance.  The  R/M 
Monitor's  work  is  not  complete  when  the  design  leaves  the  drawing  board, 
nor  when  the  equipment  Is  Installed  on  the  site.  Information  on  proper 
Installation,  operating  instructions,  and  maintenance  procedures,  both  FM 
and  CM,  must  be  available.  The  initial  review  of  these  manuals  should  take 
place  In-plant.  Check  should  be  made  for  content  and  format.  Early  eval¬ 
uation  can  be  obtained  in-plant  by  noting  use  of  the  manuals  on  the  test 
floors  and  in  the  repair  departments .  The  technicians  should  be  asked  for 
comments  concerning  the  manuals.  After  equipment  has  moved  to  the  Category 
II  test  phase,  comments  of  the  Using  and  Training  Cocmands  can  be  obtained 
for  further  evaluation  of  adequacy  of  the  manuals.  The  Contract  Management 
Regions  (Ofls)  have  specialists  assigned  to  the  district  offices  to  assist 
In  manual  and  drawing  preparation.  Their  assistance  should  be  requested. 
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n-Hcuse  Manufacturing.  Packaging,  Storing r  and  Transporting 

MIL-STD-441  establishes  the  basic  definitions  and  reliability 


i.  In- 
Procedures  . 

concepts  for  DOD.  The  prime  consideration  of  the  Using  Canmands  is  the 
"Operational  Reliability”,  that  is,  what  Is  available  to  them  when  the 
equipment/system  is  installed  on  site  and  is  In  the  field  environment. 

The  predicted  reliability  obtained  by  using  the  techniques  outlined  in 
the  reliability  specifications  such  as  MIL-R-26474,  or  the  RADC  Notebook, 
is  the  "inherent  Reliability”  which  is  the  potential  reliability  that  the 
equipment  is  capable  of  delivering.  This  potential  reliability  is  de¬ 
graded  during  the  manufacturing,  handling,  testing,  etc.,  phases  before 
delivery  and  installation.  The  R/M  Monitor  should  check  the  in-house  pro¬ 
cedures  to  see  that  proper  consideration  is  given  to  preserving  the  in¬ 
herent  reliability.  Some  points  that  should  be  checked  are: 


(1)  Receiving,  stocking,  storage,  and  issue  procedures.  Is  the 
Identity  of  equipment  properly  recorded  and  available  for  checking 
against  original  data?  Are  the  bins  for  storage  adequate  to  prevent  damage 
from  handling,  dust,  temperature,  etc.?  Are  defective  parts  properly 
logged,  tagged,  sent  to  laboratories  for  analysis? 

(2)  Manufacturing  and  assembly  process.  Are  the  assembly  details 
adequate  and  detailed?  Are  the  workers  properly  Instructed  and  qualified 
for  their  positions  such  as  welding,  soldering,  etc?  Have  the  techniques 
been  checked  to  determine  they  are  the  best  available  for  the  particular 
application? 


(3)  Process  handling.  Are  the  assembly  and  transporting  lines 
properly  designed  to  hold  and  protect  the  equipment  from  dropping,  bumps, 
and  shocks?  Are  the  assemblies  tagged  and  identified  in  the  processing 
so  that  process  control  can  be  checked  against  test  results? 

(U)  Plant  testing.  Is  the  test  equipment  adequate  and  accurate? 

Are  the  records  fully  documented,  dated,  and  verified?  How  are  the 
records  handled,  analyzed,  and  tabulated? 

J .  Spares  and  Aerospace  Ground  Equipment  (AGE) : 

(l)  It  is  interesting  to  note  that  several  cases  have  been 
observed  where  a  contractor  was  shipping  material  as  spares,  which  would 
not  meet  the  specification  requirements  of  the  Prime  Mission  Equipment 
(FME) .  This  was  a  result  of  the  contract  not  specifically  requiring  certain 
tests  on  the  spares  that  were  required  on  the  PME.  In  general  the  same 
requirements  should  be  required  for  all  spares  that  are  called  for  on  the 
"initial  buy".  The  R/M  Monitor  should  take  advantage  of  the  normal  time 
lag  in  spares  procurement  to  see  that  the  problem  areas  noted  In  the 
"initial  buy"  are  not  repeated  on  "follow-on  buys". 
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(2)  Ah  our  systems  become  more  complex  and  the  need  for  higher 
Availability  Increases,  the  specialized  AGE  requirements  become  mere 
complex.  Adequate  R/M  requirements  should  be  delineated  In  the  SOW 
and  provipion  made  In  the  R/M  plan  and  monitoring  programs  to  see  that 
the  system/equipment  needs  are  satisfied.  In  general,  the  seune  procedures 
and  attention  to  detail  should  be  given  to  the  AGE  as  Is  given  to  the 
PME.  Since  AGE  frequently  requires  calibration  not  readily  available  at 
the  operating  sites,  arrangements  should  be  made  for  calibration  and 
check-out  with  the  AF  calibration  group.  If  required  by  the  system, 
arrangements  should  be  made  for  acquisition  of  a  Precision  Measurements 
Electronics  Laboratory  thru  the  Newark  Air  Force  Station,  Ohio. 

k.  Indoctrination  and  Training.  This  element  Is  required  In  most  of 
our  R/M  specifications.  It  should  be  given  proper  consideration.  The  AF 
does  expect  R/M  to  be  considered  by  all  concerned,  from  the  design  engi¬ 
neers  through  to  the  shipper.  The  SPO  should  take  Into  account  the  fact 
that,  in  many  cases,  the  contractor  may  have  had  other  contracts  with  the 
AF  and  has  had  similar  clauses  in  his  earlier  contracts.  The  Monitor 
should  check  this  point.  He  should  try  to  determine  what  has  already 
been  covered  and  what  is  needed,  then  require  the  contractor  to  fill  the 
gap  by  completing  whatever  training  is  needed  to  give  the  SPO  the  results 
desired.  It  13  expected  that  when  lectures  or  training  is  accomplished 
by  the  contractor,  that  the  SPO  will  be  given  a  set  of  the  notes  covering 
the  lecture  or  training,  with  a  list  of  attendees  by  name  and  Job  or 
function.  The  Government  is  not  interested  in  training  the  contractors 
commercial  department  at  AF  expense . 

The  Monitor  may  suggest  several  excellent  films  which  are  available 
through  AF  sources  and,  in  addition,  the  Professional  Technical  Group  on 
Reliability  of  the  Institute  of  Electrical  &  Electronic  Engineers  will 
provide  speakers  an  request.  As  noted  earlier,  this  should  be  a  specific 
’’task"  in  systems  contracts .  If  the  manpower  and  planning  on  this  item 
is  known,  the  SPO  is  in  a  position  to  get  the  most  value  per  dollar 
expended  and  redirect,  the  effort  if  necessary. 

l.  R/M  Math  Model.  The  R/M  Math  Model  (Figure  .1)  is  In  effect  a 
tool  or  device  to  determine  from  time  to  time  the  results  achieved,  and 
where  there  are  trouble  points.  In  addition,  it  is  a  means  to  determine 
where  future  major  efforts  will  be  needed.  There  are  simple  models  and 
very  complex  ones.  The  contractor  should  develop  a  model  that  will  meet 
the  SPO  purposes  at  least  cost.  It  is  interesting  to  note  that  some 
contractors  have  "computerized”  their  models.  If  the  contract  is  large 
as  well  as  complex,  this  is  a  good  approach.  By  this  means  the  R/M  Engi¬ 
neer  can  readily  charge  certain  equipment/sys terns  parameters  and  quickly 
determine  the  overall  effect. 


1 6 


Genera: : 


a.  Use  of  Reports: 

(1)  Usually  each  of  the  elements  of  the  R/M  program  must  be  con¬ 
sidered  and  be  reported  on  regularly.  Depending  on  the  report  cycle 
which  may  be  set  up  on  a  time  cycle  basis  (monthly  or  quarterly)  or  mile 
post  basis  (Ref:  Fig  II)  the  contractor  Is  expected  to  cover  ea.n  element 
in  his  report.  The  SPO  Monitor  should  check  the  progress  on  these  items. 

As  a  convenient  means  of  checking  progress,  it  is  suggested  that  a 
Problem  List  be  developed  as  part  of  the  report.  This  list  should  provide 
a  quick  reference  of  the  status  for  the  SPO/contractor  effort.  Development 
of  a  single  uniform  problem  list  for  use  by  the  contractor  and  the  SPC 
assists  in  proper  monitoring  of  the  program.  In  this  manner  ea„h  pTObiem 
as  reported,  is  entered  into  the  list  and  remains  on  the  list  unti- 
resolved  to  the  satisfaction  of  the  SPO.  A  sample  form  for  such  a  Frcbiem 
List  is  shown  as  Figure  VIII . 

(2)  The  contractor  should  be  advised  at  the  start  of  the  contract 
what  is  expected  in  each  report  In  terms  of  content,  documentation,  and 
format . 

(3)  Reports  should  be  used  to  assist  in  monitoring  the  program 
If  the  reports  are  timely,  complete,  and  accurate,  they  will  indicate  the 
progress  made  to  date,  existing  and  potential  trouble  areas.  Th**  SPO 
can  then  plan  where  to  concentrate  its  efforts.  In  view  of  the  personnel 
shortage  it  is  important  to  use  these  reports  to  get  a  maximum  value  for 
each  dollar  or  hour  expended.  Some  SFOs  use  the  "draft  route*'  on  their 
reports,  others  are  formally  submitted  Initially,  with  corrections  to  be 
noted  ir.  the  next  iesue.  Either  method  is  acceptable.  It  ,is  important 
that  the  reports  be  reviewed  by  the  SPO  for  conformance  with  program 
requirements.  If  otranlssions  or  errors  are  noted,  the  contractor  should 
be  advised.  Normally  the  contractor  should  have  SPO  comments  00  his 
report  ten  days  to  two  weeks  after  receipt  by  the  SPO. 

(ii)  The  final  report  on  each  program  should  be  of  the  summary 
type.  It  sr.ouid  trace  the  R/M  history  of  the  program  noting  the  Important 
developments  in  the  cycle,  the  problem  areas  noted,  and  the  mean?  used  to 
resolve  them.  This  report  should  be  prepared  in  accordance  with  ESD 
Exhibit  63-I,  Volume  III  as  a  Technical  Documentary  Report  and  should 
be  sent  to  the  Defense  Documentation  Center  so  that  the  information  will 
be  available  for  ready  reference  on  new  procurements. 

b.  Records .  It  is  highly  desirable  to  incorporate  a  requirement 
for  retention  of  records,  data,  and  drawings,  etc.,  In  the  prime  contra:t 
In  addition,  the  prime  contractor  should  be  required  to  retain  data 
cover irg  subcontractor’s  work  for  a  specified  number  of  y^ars  (three 
years  after  completion  of  contract  is  suggested) .  There  should  also  be 
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a  statement  time-phasing  the  establishment  of  this  data  file.  This 
contractual-  requirement  is  needed  due  to  the  complexities  and  financial 
prohlems  involved  in  doing  business.  Cases  are  on  record  where  con¬ 
tractors  have  gone  hankrupt  either  during,  or  shortly  after  completing, 
a  contract.  The  result  has  been  that  the  AF  has  items  in  its  inventory 
without  nomenclature,  failure  rates,  replacement  drawings,  etc.  This 
has  caused  major  maintenance  and  reprocurement  prohlems. 

c .  Plant  Visits: 

(l)  There  is  no  fixed  R/M  organizational  requirement  in  Government 
specifications  or  regulations  at  this  time,  nor  weald  a  rigid  requirement 
specifying  a  certain  type  of  organization  serve  a  useful  purpose  since 
the  R/M  work  varies  from  contract  to  contract  and  is  affected  hy  the  type 
of  personnel  availahle  to  do  the  Job . 

The  R/M  requirements  of  the  specifications  are  usually 
developed  as  specific  tasks  (Ref:  ESDP  80-2)  and  incorporated  in  the  SOW. 
The  contractor  then  assigns  these  tasks  to  operational  elements  within 
his  organization  for  fulf illment .  Ideally,  the  R/M  group  should  be  set 
up  as  a  separate  organizational  element  reporting  to  top  management  on 
a  par  with  the  engineering  and  production  management  groups .  The  organi¬ 
zation  will  vary  bQmevhat  from  company  to  company  depending  on  the  type 
of  contract  (R&D  or  production)  and  whether  the  prime  contractor’s 
function  is  primarily  production  or  management  of  a  given  weapons  program. 
For  programs  of  moderate  size,  the  following  departments  would  he  directly 
involved  with  the  R/M  groups : 

(a)  Purchasing. 

(h)  Incoming  Inspection. 

(c)  Specification  Control. 

(d)  Environmental  Test. 

(e)  Quality  Control  (QC). 

(f)  Engineering  Sections  (Design,  Production,  Test). 

(g)  Field  Engineering  (Plants  and  Sites). 

Generally  there  is  a  close  tie  between  the  QC  organizations 
and  R/M  groups.  This  is  due  to  the  similar  nature  of  the  work  involved, 
especially  in  handling  subcontractor  operations  and  field  testing.  In 
some  companies,  these  activities  (R/M  and  QC)  are  combined  under  the 
heading  of  a  Product  Assurance  (PA)  Department. 
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The  R/M  Monitor  should  estahlish  contact  with  the  contractor 
R/M  or  PA  Department  early  in  the  contract  cycle.  He  should  determine  the 
degree  of  communication  hetween  the  groups  and  see  that  the  R/M  group 
pursues  their  activities  diligently.  These  contacts  should  result  in  an 
AF/Contractcr  team  approach.  A  typical  contractor  R/M  organization  is 
shown  below: 


(2)  When  visiting  the  plant,  the  Monitor  should  visit  all  depart¬ 
ments  concerned  with  the  contract.  Complete  answers  as  to  R/K  conditions 
are  not  found  i;i  the  top  management  offices.  The  place  to  get  the  infor¬ 
mation  is  in  the  factory,  ip  the  Engineering  Department,  at  the  test 
benches,  and  at  the  sites.  If  the  plant  visit  is  made  by  an  AF  group, 
arrangements  should  be  made  to  split  the  group  so  that  one  suh- group  car. 
talk  to  one  department  (i.e.,  Engineering)  and  the  other  to  another 
department,  (i.e..  Reliability).  Arrange  to  ask  each  department  the  sore 
questions ♦  The  fub-groups  should  then  compare  in  private,  or  out3i.ie  the 
plant,  the  answers  ohtained.  This  is  an  excellent  means  of  checking 
intra-plant  corcTunications .  A  similar  approach  can  be  used  hy  sending 
ore  man  to  a  pr’me  contractor’s  plant  and  one  to  a  subcontractor's,  and 
comparing  the  results  to  see  if  there  is  proper  direction  of  the  sub¬ 
contractor  by  the  prime  contractor. 
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A  re  tori  should  be  kept  of  all  findings  noted  during  these 
visits.  The  contractor  should  be  advised  of  conditions  noted  ana  actions 
required.  On  later  visits  these  records  should  be  checked  to  see  that 
satisfactory  results  ars  obtained. 

d.  Problem  Rrgclutlon.  If  a  question  or  problem  arises  with  the 
contractor  that  the  Monitor  can  not  resolve  immediately,  the  following 
action  is  suggested: 


(1) 

(2) 

and  discuss 


Be  sure  the  problem  is  clearly  understood. 

Get  any  suggested  solutions  the  contractor  nay  wish  to  offer, 
thf=m. 


(3)  DLsru-. -  any  appr-tet^hes  the  Monitor  may  have. 

(h)  Advise  the  contractor  that  the  answer  will  be  forthcoming  as 
soon  as  possible  after  discussion  at  the  SPO. 

(5)  Cr.^  k  for  the  needed  information  through  AF  sources. 

(6)  Advice  the  contractor  of  the  decision. 


Giving  a  prompt  but  unsound  answer,  or  an  erroneous  one,  serves  no  useful 
purpose.  It  may  mean  that  AF  does  not  get  good  equipment.  It  may  mean 
Increased  costs,  but  most  damaging  of  all,  it  gives  the  impression  that 
the  monitor  does  net  fully  understand  the  situation  and  can  be  led  into 
making  decision?.  This  can  have  serious  results  at  a  later  date, 

e .  Assir*a:.-,tj  Available  to  the  Monitors: 


(1)  PADC  frequently  assists  the  Monitor  from  a  technical  viewpoint 
in  evaluation  arl  review  of  math  models,  equipment  monitoring,  etc.  The  RADC 
Project  Engineer'  usually  have  responsibility  to  establish  performance 
criteria.  Ir.  "rtain  instances  RADC  is  charged  with  formal  acceptance  for 
the  Government .  Support  by  PADC  Is  established  through  use  of  ’’Charter 
Letters"  (Ref-  ESOR  60-2  and  80-4) • 

(2)  Thp  ESP  R/M  S*aff  Office  (ESTE)  Lr  available  for  assistance 
in  selected  management  an I  technical  matters.  The  ESD  Reliability  staff 
normally  reviews  RFPs,  bids,  contracts,  and  30Wo .  On  oc-aeion,  assistance 
io  available  in  ti-ider.*'  briefings,  contract  negotiations,  and  plant  visits. 

(3)  Tb^  "MFc  have  personnel  assigned  to  various  geographic  and 
technical  areas.  They  are  available  for  in-plant  monitoring,  test  verifi¬ 
cation,  subcontract  monitoring,  manual  and  drawing  reviews,  design  reviews, 
etc.  Arrangements  fer  CMR  assistance  is  Initiated  by  SPO/JMR  letters  of 


20 


agreement  and  contract  delegation  through  the  Administrative  Contracting 
Officer. 


(U)  MITKh  hau  support  agreements  with  certain  SPOs  providing  for 
engineering  assistance  In  systems  engineering  matters.  Limited  R/M  support 
is  available  under  these  agreements . 
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NOTE  1:  IN  LOCAL  OPERATION  MTBF  IS  1545 
IN  REMOTE  OPERATION  MTBF  IS  1180 

NOTE  2 :  REMOTE  CONTROL  UNIT  IS  OMITTED  FOR  "LOCAL" 
OPERATION  MODE 
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FI  SURE  IV  EXAMPLE  OF  FAILURE  REPORT  FORM 


OPERATING  TIME  LOG  Page  Number 


F'GURE  V  EXAMPLE  OF  OPERA!  NG  TIME  LOG 


i 


C.  TIME  SPEC 

O.  WK  AREA 

E.CST  M/H 

F.  OR IO  RPT  NO 

G.  REPORT  NO 

M  , 

N«Q 

h.  JOS  cdmt  no 


I  ,  «Vfc  »  t 


a.  IKNIAL  NO 


a.  tim* 


4.  MONK 
CINTi 


a.  WON  K  OKOKK  NO 


id. 'how  MAI 


a. O A  Y  MO  VK 


7.  WON  K  UNI  I 

code 


IA.  AOK  MUC 


a.  serial  no 


IB.  ENG 

PSN 

as.SER  MDO 

YR.MPO 

TM 

1 

U 

a.  ACT 
TAKE* 


a.  WHEN 

oia 


I  I.  UNIT* 


1 4#  IN»T  ENO 
TM  PIN 


I*.  I  A  B  OR 
HD  UN  S 


St  N  MOO  VN-MPG  MENIAL  NO 


IS.  AIIT 
WON  K 
CINTIm 


1C.  ITEM  r»C  I  IC.  WANT  NO 


ac.  MENIAL  NO 


I  7.  INMT  IT  EM  PT  NO 


i  a.  menial  no 


J.IVMIOL  K.  OIMCNEP ANC  Y 


L,  CONNECTIVE  ACTION 


SUPPLEMENTAL  DATA 

ai.  taik.  itant  aa.  mailed  item  T  aaT  pailio  item  a  4.  neat  ammy  aa.  neat  asTy  I  is.  next  am  my  27.  nc*i  ajsy 

no  NOUN  MPa  P/N  S/N  NOUN  MFC 


28.  OPERATIONAL  TIME  AT  FAILURE 


28.  SEVERITY  OF  FAILURE 


10.  TYPE  FAILURE 


nKI  MIN 
MEL 


CAL  TIME 

C 


MILES 

O 


CRITIC 

A 


MAJOR 

B 


MINOR 

c 


PRIM 

A 


SEC 

B 


31.  ANAL 

REQ 


YES 

A 


NO 

e 


32.  FAILEO  ITEM  DISPOSI¬ 
TION 


FIELD  ItMVICE 
B 


3*.  Y.  6.  proced 

NO 


IN  AOEO 
B 


if.  USE  OF  SPECIFIEO  TOOLS 


33.  MAINTENANCE  TIME 


34,  NO  PERSONS  USED 


35,  EQUIV  A F SC 


TOTAL  ODWN  TIME 

A 


TIT 


ACTUAL  REPAIR 
TIME 


TNOUILE  REPAIR 

SHOOT 


39.  FOLWOW-UP  AQKNCY 

□  OC  □  ENO  I  i  OTHER 


AOEO 

A 


IN AOEO 

8 


NOT 

NEEDED 

c 


4a.  af>contrac  TOR  USE 


DISC  OVER  to  EY-i  10  NATURE  S 

OR  ADE 


CORRECTED  BY.SIGNATURC  t  SHADE 


INSPECTED  B  Y*  SIGNATURES 
O  R  AO  E 


SUPER  VISOR* SIGN  ATURE  » 
OR  ADE 


OAT*  TRANSCRIBED  I  TRANSCRIBED  BY-SIONATURE  &  GRADE 


a  UNCLEARED  I - )  R  E  P  L  AC  EMC  NT  T  II 

DISCREPANCY  ‘ - 1  CHANGE  ITEM 


□  CRIB 
APP 


DATA  TRANS- 
BED  TO 
RDP 

RECORDS 


FIG  VI-  MAIHTENANCE  DISCREPANCY/PRODUCTION  CREDIT  RECORD 

PRECEDING 
PAGE  BLANK 


33 


20.  PANTS  REPLACEO  DURINO  REPAIR 


FIGURE  V 


l-A  MAINTENANCE  DISCREPANCY/PRODUCTION  CREPIT  RECORP 


Re  I  iobi  I  ity/Mointa  inability  Modification  Analysis  Report 
Equipment: 

Description  Drawing  No. 

Manufacturer 

S/N 

Propose  to  Start  with  Equipment  No. 

Estimated  Cost/tlnit _ 

Modifications  (list  and  explain;  attach  drawings) 

1. 

2. 

3. 

4. 

Analysis 


Condition 

Value* 

Modification 

(See  Modification 

(MTBF)  Figure  of  Merit 

Na .. 

Analysis  Chart) 

R  '  R 

Remarks 

(') 

(2) 

1 

1 

•Note;  Attach  all  data  and  computations  (See  Reverse) 


FIGURE  VII  EXAMPLE  OF  MODIFICATION  ANALYSIS  SUMMARY  FORM 
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R/M  Quo  Ji  to  five  Modlficotion  Analytli  Chart 

CONDITION 

RELIABILITY 

maintainability 

A 

Favorable 

Favorable 

B 

Favorable 

Advene 

C 

Advene 

Favorable 

D 

Advene 

Advene 

E 

No  effect 

Advene 

F 

No  effect 

favorable 

G 

Advene 

No  effect 

H 

Fovorabfa 

No  effect 

1 

No  effect 

No  effect 

FIGURE  VI I- A  MODIFICATION  ANALYSIS  CHART 
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FIGURE  VIII  EXAMPLE  OF  CORRECTIVE  ACTION  FOLLOW-UP  REPORT 
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Development  of  Circuit/Ry3tan  Mathematical  Model 

1.  The  essential  steps  in  development  of  a  mathematical  model  are  given 
belcw*  While  the  procedure  given  deals  with  relatively  simple  config¬ 
urations,  the  principles  involved  also  apply  to  the  most  complex  equip¬ 
ment  /'•ysteu.3.  Attachment  1  gives  the  Reliability  Functions  for  Various 
Active-Parallel  Configurations . 

.•tec  1  - 

Define,  the  equipment/system  in  terms  of  its  required  functions. 

. tep  2  - 

Construct  the  Reliability  Block  Diagram.  To  do  this,  the  equip¬ 
ment  /system  must  be  analyzed  in  terns  of  its  parts  (subordinate  eler.ents) 
shewing  the  "Reliability  Paths"  through  which  the  desired  function  will 
be  accomplished • 

.jtep  3 

Describe  the  reliability  block  diagram  in  terms  of  its  success/ 
failure  probabilities.  Separate  statements  should  be  developed  for  each 
path  cr  mode  of  operation. 

.  tep  U  - 

.-si..’  ate  the  Reliability  of  the  individual  parts  or  elements  in 
^eiiabili  lj;  bloc  diagram.  This  can  be  based  on  standard  failure  rates 
(.viTC  notebook),  results  obtained  with  similar  equipment,  cr  actual  test 
result-*.  Rad  lure  rates  should  be  adjusted  based  on  the  environmental 
application  and  stress  level1*  used. 

5  - 

Obtain  the  numerical  value  (Probability  of  success  (P))  of  the 
equipment /system. 

d.  The  following  points  should  be  noted  in  the  development  -and  application 
-  f  the  mathematical  model* 

.•>.  j  open  ding  cn  the  accuracy  of  the  results  desired  lew  failure  rate 
; .  a  ,  such  as,  tube  sockets,  ten.'.inal  strips,  etc.,  may  be  omitted  from 
the  calculations* 

b.  equiprent/aystw  can  operate  ?n  alternate  moie3,  these  must 
be  ccn^iiorei  separately. 


c.  lensi deration  must  be  given  to  types  of  redundancy  involved 
(active,  *tmi-by,  etc). 


i.  If  the  overall  equipment  /system  ltd  lability  value  *inally  obtained 
dees  net  reel  require. ents  changes  can  be  made  by  studying  the  "weak  links." 
These  may  be  improved  by  adding  redundant  eler.entr.,  additional  lerating, 
v s : r. ^  improved  parts,  etc. 
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e.  The  sane  type  of  analysis  can  be  used  for  evaluation  of  other 
system  parameters,  such  as,  ccst,  volume,  weight,  effectiveness,  etc. 

f.  In  development  of  failure  rates,  certain  "K"  factors  may  be  needed 
to  compensate  for  environments,  such  as,  airborne  vs  ground-operated  equip¬ 
ment  or  changes  in  duty  cycles. 

3.  The  basic  system  configurations  are  shown  below.  All  equipment/ systems 
can  be  resolved  into  similar  configuraticns  for  problem  solution.  In  the 
examples  given,  the  following  notations  are  used: 

probability  of  success  or  reliability  of  a  system  cr  blcck. 

r  *  probability  of  success  or  reliability  of  a  unit  or  path. 

Pi  «  probability  of  success  or  reliability  cf  element  i. 

qi  ■  probability  of  failure  or  the  unreliability  of  element  i. 

(NOTE:  Pi  +  qi  »  1.0) 

Elements  of  a  system/equipment  are  designated  as  A,  B,  C,  etc.  For  prcba- 
bility  statements  concerning  element  success  or  failure,  the  fcllcwing 
notations  are  used: 

(A)  =  the  event  or  success  of  element  A. 

P(A)  =  probability  that  event  A  occurs  pa. 

ij  **  the  element  in  the  i  th.  row  and  J  th  column  where 
i  =  1,  2,  etc;  j  -  1,  2,  etc. 

oq 

p(t)  =  element  reliability  function  **  Jt  (t)  dt 

t 

q(t)  =  1  -  p(t)  —  element  unreliability  function. 


//hen  elements  have  an  exponential  failure  density  with  failure  rate  X  , 
then  the  reliability  function  p(t)  *  e  and  the  unreliability 

function  =  q( t )  -  1  -  e 

a.  Series  System 


Rel  system,  *  P(AX)  .  P(A2)  ...  P( 4^) 


Where  P  *  Probability  that  A  will  operate.  In  this  system  to 
achieve  success,  all  elements  must  operate. 
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Parallel  System 


Input 


Output 


Rel  System  f  P(A)  +  F(B)  -  P(A)  .  ?(E)  = 

In  this  system,  success  is  achieved  if  A  and/or  B  is  cperable. 
3.  Peries -Parallel  System 


s  “  [p(A)]  •  [t(n)  +  r(C)  -  P(B)-P(C)J 
In  this  system,  tp  be  successful,  A  must  be  operable  plus'B  and/or  C» 
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RELIABILITY  AND  MAINTAINABILITY  PROGRAM  PROBLEMS 
OBSERVED  DURING  CONTRACTOR  MONITORING 


FOREWORD 


T.  e  purpose  of  tr.is  section  is  to  set  forth  twelve  major  problems 
observed  by  the  OPR  and  SPO  R/M  Engineers  during  the  course  of  monitoring 
contractor  R/M  Programs. 

Tnese  problems  were  observed  over  the  past  year  and  a  naif.  In  each 
case,  tr.e  SPOs  involved  have  required  corrective  action  by  the  contractors. 

The  contents  of  this  section  serves  as  a  guide  for  new  SPO  R/M  Engineers 
in  tne  prevention  of  similar  problems  on  their  program. 


f 


SECTION  VIII 

CONTENTS 

SUBJECT  PAGE 

1.  Introduction . v .  1 

2.  R/M  Program  Problems  . .  1 


iii 


RELIABILITY  AND  MAINTAINABILITY  PROGRAM  PROBLEMS 
OBSERVED  DURING  CONTRACTOR  MONITORING 


Contractor  R/M  programs  conducted  in  accordance  with  existing  Air  Force 
specifications  require  e  vigorous  monitoring  by  the  involved  SPOs . 

During  the  past  year  end  e  he  If ,  the  OPR  and  various  SPO  R/M  engineers 
hare  been  complying  with  this  Air  Force  requirement.  This  activity  has 
served  to  identify  certain  contractor  practices  which  must  be  avoided,  if 
R/M  programs  are  to  be  successful. 

Twelve  of  these  practicee  are  identified  in  paragraph  2  below.  New  SPO 
R/M  engineer..  will  take  the  neceseary  action  to  avoid  their  occurrence  on 
future  SPO  programs. 

2.  R/M  Program  Problems: 


a.  Lack  of  a  Well -Organ 1  zed  Equipment  Dealgn  Review  Effort.  Currant  R/M 
programs  requixa  periodic  formal  design  reviews.  The  main  purposa  of  such 
reviews  is  to  detect  and  eliminate  potential  causes  of  unreliability  and/or 
unmaintainability  early  in  the  program.  The  cost  involved  in  making  changes 
during  the  design  phase  of  e  program  has  been  estimated  to  be  1/1000  of  the 
cost  involved  efter  equipment  delivery  to  the  Air  Force.  A  vigorous  engineer¬ 
ing  design  review  effort  must  be  accomplished  by  contractors. 

This  effort  involves  identification  of  the  contractor  personnel  .and/or 
organizations  Involved  in  design  reviews ,  responsibilities  and  authorities  of 
a  Design  Review  Board,  frequency  (of  design  reviews),  and  approach  to  conduct 

of  reviews. 

A  Board  is  expected  to  maintain  mmutea  of  meetings  end  to  essign 
corrective  action  follow-up  responsibilities. 

Reviews  of  contract  r  ^ M  pro grama  have  served  to  identify  inetancea 
of  no  company  policy  on  ds_-  gi<  J  and  no  organizational  reaponeibility 

for  dea.gn  reviews.  A  systematic  approach  to  design .reviews  abould  Involve 
the  use  of  a  design  review  checxllat.  The  absence  of  such  a  checklist  waa 
noted  In  almost  all  the  programs  revis'd 

b .  F ailura  to  lni<_?rE*; rate  _Qij ant  i *  ve  JB n \  tab  *  1  itv  Requirements  In 
SuIka  nti actor  Specifications  Th*  attainment  of  a  apecific  quantitative 
relieblllty  level  in  a  complei  iy  item/squl  pment  requires  that  a  prime 
contractor  specify  firm  quantitative  requirements  in  subcontractor  equipment 
specif  lotions . 
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Paragraph  3-5*5  of  MIL-R-27542A  (USAF)  states  that: 

"The  contractor  ahall  Impose ,  directly  or  Indirectly,  quantitative 
reliability  requirements  and  acceptance  criteria  on  ail  echelons  of  suppliers 
and  subcontractors. 

"The  contractor  shell  take  ell  actions  neceseery  to  assure  that  no 
changes  made  by  any  supplier  will  reduce  reliability  of  the  system." 

c.  SpeaBodlc  Cowmmlcatlon  Betv.cn  R/ft  and  Design  Engineering.  Frequently, 
contractor  R/M  organizations  have  seemed  tc  operate  independently  of  other 
contractor  organizations,  such  es,  design  engineering^  e.g.,  basic  recommenda¬ 
tions  on  application  of  piece  parts  to  minimize  failure  rates  have  been  found 

to  be  sketchy.  No  established  channels  of  communication  for  informal  discus¬ 
sions  on  R/M  design  problems  3eemed  to  be  the  rule. 

The  communication  prohlsm  la  considered  to  be  e  management  rather 
than  a  technical  consideration .  Unless  contractor  management  takes  action  to 
foster  such  communication,  independant  or  quasi -independent  operations  will 
continue  to  the  detriment  of  Air  Force  product  reliability. 

d.  Poor  Corrective  Action  Procedures.  Ahaence  of  well  coordinated 
corrective  action  efforts  have  been  observed.  Identified  R/M  problems  have  not 
been  properly  scheduled  for  corrective  action.  Follov-up  to  asaure  problem 
reaolutlon  vas  lacking.  In  certain  cesea,  system/ equipment  engineers  were  not 
even  svare  of  the  existence  of  R/M  problems.  Failure  data  analysis  by  Itself 
will  not  Improve  equipment  R/M.  It  is  necesaery  to  supplement  such  analysis 
with  a  corrective  action  effort  which  assures  that  identified  R/M  problems 

or  "week-links"  are  reaolved. 

e.  Late  Reports.  Month Ly  reports  on  R/M  have  been  found  to  be  as  much  es 
six  weeks  lete.  The  result,  of  course,  is  that  SPOa  have  late  information  for 
decision-making  on  the  progress  of  R/M  programs 

Fur therm*  re ,  contents  of  reports  ha/e  been  found  to  be  of  questionable 
R/M  management  value,  e.g.,  "weak* link"  tavlea,  with  corrective  ection 
schedules^,  have  been  ncticeable  by  their  absence.  Trend  curves  which  indicate 
equipment  KTBF  "growth"  have  not  beeu  presented, 

f.  La ck-Ql ."Ground -Rules"  f  >r  Failure  Data  Beduy^lon.  Several  system/ 
equipment  programs  wre  found  t.*  have  quantitative  relishility  requirements 
txit  no  agreements  as  to  failure  definitions  or  metbe-dfs)  of  data  reduction. 
Furthermore,  it  was  net  clear  as  to  whether  or  net  the  quantitative  require¬ 
ments  referred  to  inherent  cr  operational  reliability. 

In  other  words,  the  ln/ul/od  SPQs  had  no  uleer-cut  process  of 
daterminlng  compliance  or  lack  of  compliance  with  qusntl tetlve  requirements. 
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The  specification  of  numbers  without  corresponding  deta  collection  end 
reduction  ground-rules  is  considered  to  be  en  unsatisfactory  arrangement. 

b  Vague  Basea  for  Trade-offs.  Frequent  references  to  trade-offs  between 
reliability  and  maintainability  ays tem/equi pawn t  charac teristica  without  a 
determination  of  quantitative  effects  of  auch  ectiona  appeared  es  e  consistent 
pattern  on  programs  which  incorporated  quantitative  aveilehillty  requirements. 

h.  Lack  of  Quantitative  Data  on  ECPs .  Computations  of  the  quantitative 
effects  of  proposed  changes  on  system /equipment  R/M  capability  were  found  to 
be  deficient.  In  several  situatlona,  when  quantitative  effects  were  presented, 
bac*-up  data,  and  mathematical  techniques  were  ahsent. 

In  several  caaes.  the  contractor's  R M  organization  was  not  aware  thet 
en  ECP  waa  processed  or  wa«  trng  processed. 

I.  Unsatisfactory  Mathematical  Models.  Examples  of  contractors  assuming 
e  simple  aeries  reliability  modal,  when  tho  system /equipment  contained  alter¬ 
nate  modes  and/or  redundant  replacements,  have  bean  ohaerved .  Thia,  of 
courae,  led  to  inaccurate  repreaentationa  of  equipment  capability.  While 
simplicity  in  mathematics  la  deairable,  validity  ia  considered  eaaentiel. 

J.  Unsatisfactory  R/M  Predictions.  As  e  design  progresses  and  es  failure 
and  repair  data  is  collected  and  processed,  predictions  of  aystem/equipcent 
R/M  cepebility  ahould  involve  less  assumptions.  For  example,  initially  it  is 
necessary  to  make  assumptions  es  to  how  parts  ere  to  be  operated,  and  the 
atreaa  levels  to  be  experienced.  As  the  design  progresses,  information 
becomes  available  as  to  actual  design  margins  of  safety  which  influence  the 
failure  rates  of  parts  Later,  it  ia  poaalhla  to  gather  information  on  « 
actual  streaaaa  from  engineering  hreadboerd  and  environmental  tests  which 
again  influence  feilura  rates. 

Certain  prediction  reports  have  been  reviewed  where,  after  aeverel 
years  of  wnrk  on  s  program,  assumptions  were  being  made  about  part  margins 
of  safety  and  operating  environments.  Furthermore,  final  prediction  reports 
have  been  studied  where  the  contratl.^r  continued  to  assume  a  mission  profile. 

In  *>ther  words,  the  involved  contractors  failed  to  take  advantage  of 
Increase  information  to  modify  and  updete  their  prediction  techniques 

k •  bill.  Subciaalpnof  R/M  .Program  Plqn.  While  contracts  have  called  for 
submission  tf  program  plana  30  or  45  daye  from  award  of  contract,  submittal 
dates  have  been  ignored  in  large  number  of  cases  Also,  upon  receipt  of 
proposed  plans,  It  was  noticed  that  aeveral  SPOa  had  not  given  official 
approval  of  the  plana  following  SPO  review.  Failure  to  give  auch  approve! 
leaves  contractors  lu  doubt  es  to  wheUiec  or  not  to  proceed  with  the  R/M 
programs . 
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1.  Lack  of  Coordination  Between  R/M  and  Logistics  Groups.  Failure  of 
these  activities  to  coordinate  their  work,  results  in  improper  support  of 
the  system  from  a  spares,  AGE,  and  maintenance  viewpoint.  If  R/M  does  not 
assist  the  logistics  group  in  determining  levels  of  support  needed,  based 
on  failure  and  repair  rates,  the  usual  result  is  improper  loading  of  these 
aspects  of  the  system.  Parts  are  procured  on  "percentage  basis"  and  have 
no  relation  to  actual  consumption  rates  effected  by  good  or  poor  design 
factors.  On  continuously  operating  systems  such  as  those  developed  by  ESD, 
the  preventive  maintenance  cycle  requirements  are  a  key  factor  in  deter¬ 
mining  availability.  Lack  of  coordination  results  in  "poor  guesses"  rather 
than  "good  models." 


SECTION  IX 


GUIDANCE  ON  INCENTIVE  CONTRACTING 
FOR  RELIABILITY 
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SECTION  IX 


guidance  on  incentive  contracting  for  reliability 


FOREWORD 


The  purpose  of  this  section  is  to  provide  guidance  to  SPO  Reliability 
and  Maintainability  (R/M)  Monitors  on  the  use  of  contract  incentives  for 
reliability. 

Identified  are  several  technical  problems  that  arise  when  developing 
reliability  incentives;  some  suggested -solutions  are  provided. 
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1 .  Introduction: 


Communications  received  from  Hq  Systems  Command  (see  Attachment  1)  have 
f-xpressed  a  concern  over  the  general  inadequacy  of  contractual  proviaione  for 
the  assurance  of  system/equipcpent  reliability.  Other  correspondence,  partic¬ 
ularly  directives  emanating  from  DOD,  have  specified  the  use  of  incentive -type 
contracts  in  lieu  of  the  traditional  cost -pi us -fixed -fee  (CPFF)  contrscts  for 
procurement  of  weapons  systems.  The  purpose  of  this  document,  then,  is  two¬ 
fold,  as  follows: 

a.  To  provide  tha  ESD  SPOs  with  guidance  on  the  inclusion  of  certain 
contractual  provisions  which  emphasize  contractor  obligations  associated  with 
attainment  of  specified  system/equipment  reliability. 

b.  To  provide  general  guidelines  for  stipulating  reliability  incen¬ 
tive  requirements  and  identify  certain  technical  problema  that  arise  in  tha 
process  of  doing  so.  Suggested  solutions  for  some  of  these  problems  are  al30 
provided . 

In  discussions  which  encompass  more  than  one  area  or  specialty,  a.g., 
reliability  engineering  and  contract  administration,  there  usually  arises  a 
rred  lo  establish  a  common  communications  in  order  to  avoid  misinterpretation 
of  peering  of  words.  To  .  inimize  the  problem  in  the  discussion  to  follow, 
therefore,  when  a  word  or  phrase  which  ie  unique  to  one  specialty  is  used, 
it  will  be  followed  parenl heticclly,  whenever  necessary,  by  the  appropriate 
expression  unique  lo  the  other  specialty. 

Reliability  Clause  For  Firm  Fixed  Price  Contracts: 

ESD  policy  is  that  all  system/equipment  contract  a  will  contain  minimum, 
numerical  reliability  requirements  (minimum  acceptance  reliability);  these 
requirements  are  usually  expressed  in  terms  of  mean-time-between-fsilures  (MTBF). 
Additionally,  it  is  ESD  policy  that  contractual  provisions  will  include  tha 
requirement  for  the  demonstration  of  the  stated  quantitative  requirements  at 
selected  program  milestones. 

These  policies  are  based  on  the  requirements  of  AFR  80-5,  Reliability 
Program  For  Systems,  Subsystems,  and  Equipments,  dated  24  August  1%4. 
Specifically,  paragraph  4*b  of  AFR  80-5  states  in  part:  "Specifications, 
exhibits,  work  statements,  product  descriptions,  and  contracts  for  systems 
and  associated  material,  including  Government  Furnished  Equipment  (GFE)  for 
inventory,  will  include  specific  minimum  acceptance  reliability  requirements 
es  one  of  +he  major  engineering  factors." 

In  addition,  paragraph  4-f  states:  "Rsliability  tests,  evaluations,  or 
neas  lrements  will  be  conducted  under  conditions  specified  by  the  proposal  or 
any  subsequent  test  plan  approved  by  competent  authority.  If  contractual 
reliability  requirements  ere  not  met  during  the  demonstration  teats,  the 

deficient  portions  of  tha  system  shall  be  redesigned  at  no  additional  contract 

rest  and  the  deoonstrst ion  teats  continued  or  repeated  to  verify  that  acceptance 
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reliability  bat  been  achieved.  In  the  event  uuch_ corrective  ection  proves 

Impossible  or  Impractical,  consideration  shall  be  given  to  assessing  monetary 

penalties  end  unit  price  decreases  or.  alternately.  Instituting  default  ectlon 

under  the  contract . " 

Insomuch  es  AFR  80-5  requires  of  procuring  activities  that  appropriate 
reliebility  requirements  be  included  in  contrectual  documentation,  it  remains 
for  the  SPO  to  assure  that  these  requirements  in  fact  are  Included.  This 
requires  that  numericel  requirements  end  demons tret ion  of  these  requirements 
ere  clearly  stipulated;  it  requires  that  alternatives  are  provided  which 
specify  courses  of  ection  upon  non-compliance,  i.e.,  "rejections  ard  retest". 
Although  the  extent  to  which  the  legal  responsibilities  of  e  contractor  (con¬ 
cerning  non -compile nee)  have  been  questioned  by  some  SPO  personnel,  Armed 
Services  Procurement  Regulation  (ASPR)  Section  VII  provides  us  with  provisions 
for  inclusion  in  the  Generel  Provisions  of  e  Supply  Contract,  namely:  Para¬ 
graph  5-Inspection;  Paragreph  11 -Default;  end  Paregreph  12-Disputes.  These 
three  paregraphs  are  incorporated  In  Attachment  2.  Briefly,  Paragreph  5- 
Inspection,  states:  "(a)  All  supplies  (which  term  throughout  this  clause 
includes  without  limitation  rew  materiels,  components,  intermediate  essemblies, 
end  end  products)  shall  be  subject  to  inspection  end  test  by  the  Government — 
including  the  period  of  manufacture,  and  in  eny  event  prior  to  acceptance. 

(b)  In  case  any  supplies  or  lots  of  supplies  ere  defective  in  materiel  or  work- 
car.ship  cr  otherwise  not  in  conformity  with  the  requirements  of  this  contract, 
the  Government  shall  have  the  right  either  to  reject  them— or  to  require  their 
correction.  Supplies  or  lots  of  supplies  which  heve  been  rejected  or  required 
to  be  corrected  shall  be  removed  or,  —  corrected  in  place  by  end  et  the 
expense  of  the  contrector  — .  (c)  The  Government  reserves  the  right  to  charge 

to  the  contractor  any  additional  cost  of  Government  inspection  end  test — when 
reir.spection  or  retest  is  necessitated  by  prior  rejection."  It  is  the 
responsibility  of  the  SPO  Contrecting  Officer  to  include  the  ebove  ASFR  clauses 
in  the  contracts  the  SPO  R/St  Monitor  should  take  e  personal  interest  to 
essure  that  they  ere  included. 

3.  Considerations  For  The  Reliability  Inceptive: 

As  indicated  in  paragreph  1,  it  has  become  epparent  that  more  end  more 
emphasis  is  being  placed  on  the  "incentive"  type  of  contract  in  lieu  of  the 
cost -plus -fixed -fee  type.  Severel  DOD  guidence  documents  have  been  published 
on  the  use  of  incentive  contracting.  Although  that  which  follows  pertains  to 
.r.cer.tive  contrecting  in  general,  it  provides  the  basic  concepts  by  which 
cor.sideretior  is  established  for  reliebility  incentives. 

DOD  Incentive  Contracting  Guide  (AFP  70-1-5)  provides  this  reason  for  the 
increased  emphasis  on  incentive  contrecting:  "From  1953  to  1961  the  dollar 
value  of  missiles  end  electronics  increased  from  12  to  52  percent  of  total 
hardware  deliveries.  The  reason? — a  rapidly  changing  technology  end  expanding 
requirements  for  even  more  complex  weapons  had  radically  altered  the 
charecter  and  function  of  military  research  and  development.  No  longer  the 
low-cost  predecessor  of  large,  relatively  stable  production  runs,  R&D  had 
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become  e  substantial  cost  factor  in  the  evaluation  of  every  weapons  system." 

To  cerry  this  statement  further,  it  seems  to  have  become  the  rule  et  ESD  that 
the  system  (hardware)  that  is  evolved  during  an  PJcD  effort  becomes  I£E  system 
that  is  specified  for  operational  deployment;  hence,  the  principle  of  con¬ 
currency.  Therefore,  the  initial  design  must  be  the  right  one  since  there  is 
no  "second  chance"  to  do  things  better,  including  increasing  the  degree  of 
reliebility,  except  by  way  of  ECP. 

"The  incentive  principle  holds,  in  brief,"  so  states  the  DOD  guide,  "that 
e  contrector  should  be  motivated  in  calculable  monetary  terms,  (l)  to  turn  out 
e  product  that  meets  significantly  advanced  performance  goals,  (ii)  to  improve 
on  the  contract  schedule  up  to  and  including  final  delivery,  (iii)  to  eub- 
stantielly  reduce  the  costs  of  the  work,  or  (iv)  to  complete  the  project  under 
e  weighted  combination  of  some  or  all  of  these  objectives."  The  litereture  is 
quick  to  caution,  however,  that  the  performance  incentive  (including  relia¬ 
bility)  will  not  be  used,  under  any  circumstances,  es  e  substitute  for  e  clear 
definition  of  the  desired  end  item.  It  is yiot  intended  to  place  in  the  hands 
of  the  contractor  or  the  SPO  broad  trede-off  decisions  regarding  the  final 
performance  outcome  of  the  program.  What  It  does  require  is  the  precise 
specification  of  the  nominel  performance  results  that  ere  desired.  In  this 
light,  the  performance  incentive  emerges  not  as  a  means  of  turning  the  program 
into  a  "profit  game"  opereting  between  very  wide  limits,  but  simply  es  en 
inducement  to  the  contrector  to  meet  or  exceed  the  nominal  performance  require¬ 
ments  set  forth  by  the  SPO.  In  fact,  where  the  achievement  of  target  (minimum 
ecceptarce)  performance  is  of  extreme  importance,  intentional  trede-offs  that 
sacrifice  performance  for  the  seke  of  cost  savings  should  be  prevented. 
Additionally,  a  contrect  clause  should  be  included  that  provides  for  the  loss 
of  all  eerned  cost  rewards  unless  performance  meets  or  exceeds  target  (minimum 
acceptance)  levels. 

In  evaluating  the  ebove  es  to  the  effects  on  providing  Incentives  for 
reliability  es  e  part  of  the  overall  CPIF  contract  structure,  several  fectors 
must  be  considered: 

a.  Reliability  incentives  should  be  used  as  an  inducement  to  the 
contractor  to.  meet  or  exceed  the  required  degree  of  reliability.  In  accom¬ 
plishing  this  end,  it  may  require  that  the  contractor  provide  more  conservative 
design  through  circuit  simplification;  utilize  greater  margins  of  sefety  for 
parts  application  end/or  high  reliebility  (MINUTEMAN)  parts;  or  epply  simple 
redundancy.  Reliability  incentives  require  that  studiee  must  be  performed, 
therefore,  to  determine  the  levels  of  reliebility  required  to  satisfy  mission 
requirements;  whether  these  levels  are,  in  fact,  attainable  within  design 
state-of-the-art;  and  what  the  potential  operational  support  sevings  would  be 
if  hardware  reliebility  were  greater. 

b.  The  use  of  reliability  incentives  should  provide  e  contractor  some 
degree  of  flexibility  by  allowing  for  trade-offs  between  various  performance 
parameters.  This  would  require  the  contractor  to  optimize  on  each  parameter 
in  order  to  arrive  at  the  highest  potential  incentive.  Suppose,  as  e  simple 
illustration,  that  a  certain  SPO  mission  requires  a  level  of  ays tern  "Availa¬ 
bility"  which  can  be  satisfied  with  reliability  (MTBF)  of  60  hours  and 


3 


maintainability  (MDT)  of  tvo  hours.  It  is  knovn  as  a  result  of  study  that 
each  parameter  is  the  optimum  that  can  be  achieved  by  application  of  existing 
state-of-the-art  design  techniques.  These  values,  then,  would  be  specified 
as  the  minimum  acceptance  values  (or  target  values).  Further,  suppose  that 
the  SPO  can  determine  vitb  some  degree  of  accuracy,  what  operational  support 
savings  could  be  realized  during  the  useful  life  of  tbe  system  if  the  tfTBF  were 
higher  and/or  the  MDT  lower,  and  is  willing  to  share  the  savings  with  tbe  con¬ 
tractor.  The  contractor  would  then  be  motivated  to  exploit  his  technical 
resources  to  design  a  system,  optimizing  on  KTBF  and  MDT  above  minimum  require¬ 
ments  to  the  point  where  his  additional  design  and  production  costs  equal, 
or  are  lees  than,  the  incentive  (fee)  return.  Of  course,  there  are  those 
values  of  MTBF  and  MDT  which  become  economically  impractical  (because  of  budget 
limitations)  for  tbe  SPO  to  pursue.  Additionally,  development  costs  may 
exceed  the  potential  support  savings  (see  figures  1  and  2).  These  values, 
essentially  plateaus,  would  be  used  by  the  SPO  in  structuring  the  incentive 
plan . 


c.  As  is  the  case  in  any  type  of  contract,  the  use  of  incentives  for 
reliability  requires  that  extreme  care  is  exercised  in  the  specification  of 
the  quantitative  requirement  (minimum  acceptance;  target)  and  each  objective 
level.  Definitions  must  be  provided  for  such  terms  as  malfunction,  failure, 
and  operating  time,  both  at  the  equipment  and  system  levels,  as  applicable. 

In  addition,  it  must  be  clear,  and  mutually  agreeable  to  botb  the  SPO  and  the 
contractor,  as  to  how  the  test  data  is  to  be  collected,  who  is  to  collect  it, 
who  is  to  reduce  the  data,  and  how  it  will  be  reduced.  It  must  also  be 
initially  agreed  who  is  to  operate  the  equipment,  what  the  test  conditions 
will  be,  and  how  many  equipment  /systems  are  to  be  in  the  test  sample. 

d.  Demonstration  of  attained  levels  of  reliability  presents  diffi¬ 
culties  which  are  somewhat  unique  to  the  use  of  incentives.  Whereas  a 
fixed-price  type  of  contract  may  require  the  application  of  a  relatively 
simple  sequential  test  to  demonstrate  the  "minimum  acceptance  reliability" 

(ore  value),  demonstration  of  reliability  in  an  incentive  contract  requires 
a  degree  of  flexibility  to  allow  for  the  determination  of  having  achieved  a 
certain  reliability  value  within  a  specified  range  of  values.  The  problem  is 
compounded  further  when  one  is  confronted  with  having  to  demonstrate,  with  a 
relatively  higb  degree  of  statistical  assurance,  the  reliability  of  a  multi- 
mod  ed  command  and  control  system  rather  than  a  simple  piece  of  equipment. 
Paragrapn  5  further  discussed  the  problems  associated  with  demonstration. 

4.  Reliability  In  Multiple  Incentives: 

Traditionally,  multiple  incentives  have  encompassed  three  major  areas  of 
concerns  performance;  costs;  and  schedules,  where  the  total  incentive  fee  is 
apportioned  in  some  manner  to  each  major  factor.  The  purpose  of  combining 
incentives  is  obvious.  Successful  performance  of  almost  any  contract  consists 
of  completing  a  satisfactory  end  item  or  service  at  a  reasonable  cost  and 
vlthir  certain  time  limits.  Since  all  these  factors  are  closely  dependent  on 
esch  other,  a  contract  that  places  too  heavy  a  premium  on  one  risks  a  loss  of 
control  over  the  other  two.  It  follows,  then,  that  a  properly  structured 
multiple-incentive  contract  should  serve  two  basic  pur poses* 
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Cost  (Normalized) 


fielatippshjp?  between 
Cost  for  Reliability  and  OperatjnE_Cgatg 


MIBF  (Normalized) 
Figure  1 


The  total  cost  of  a  system  is  tbe  sum  of  initial  design-for- 
reliability  investment  and  operational  costs.  The  opposite 
slopes  of  these  two  factors  produces  the  minimum  cost  "saddle* 
in  the  total  cost  curve. 
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Increasing  Costs 
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.Relationship  Between  Total  Costs 

and  Level  of  Reliability 


Incentives 


Figure  2 


Too  little  or  too  much  investment  in  initiel  reliability  can 
be  expensive  when  considered  on  the  basis  of  total  system 
cost  (includes  both  procurement  and  support  costs).  Incertives 
are  warranted  only  in  those  ereas  where  the  present  reliability 
of  the  type  of  system  can  be  economically  improved. 
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a.  It  should  motivate  the  contractor  to  strive  for  outstanding 
results  in  all  three  incentive  areas.  In  other  words,  his  objective  at  the 
outset  should  be  to  earn  maximum  profit,  and  the  contract  should  be  structured 
so  that  there  is  some  possibility  that  he  can  do  this. 

b.  The  incentive  structure  should  compel  decisions  between  cost,  time, 
and  performance  that  ere  consistent  with  the  overall  procurement  objectives 

of  the  SPO  when  it  becomes  apparent  to  the  contractor  that  outstanding  results 
cannot  be  achieved  in  all  areas. 

Realization  of  the  first  objective  depends  largely  on  the  range  of  effec¬ 
tiveness  (minimum  acceptance  to  maximum,  or  objective,  values)  established  for 
each  incentive  element  end  the  probability  of  achieving  outstanding  performance 
in  all  incentive  areas.  On  the  other  hand,  realization  of  the  second  purpose 
is  based  mainly  on  the  relative  weights  assigned  to  each  incentive  element 
since  weights,  along  with  the  separate  ranges  of  effectiveness,  will  establish 
the  various  break-even  points  for  trade-off  decisions  between  cost,  schedules, 
and  performance.  A  major  consideration,  then,  is  how  these  two  factors — 
relative  weighting  and  range  of  effectiveness — are  to  be  determined  for  a 
given  situation,  and  more  specifically,  the  manner  by  which  the  incentive  for 
reliability  is  established. 

Just  as  the  minimum  acceptance  reliability  requirements  must  be  established 
in  e  fixed -price  contrect,  so  is  the  case  when  incentives  are  contemplated. 

In  fact,  when  the  range  of  effectiveness  is  developed,  the  lower  limit  of  the 
range  is  set  by  the  SPO  as  being  e  minimum  acceptance  reliability  value  which 
will  be  satisfactory  even  if  a  contractor  does  poorly  on  this  incentive 
element.  Therefore,  care  should  be  taken  to  set  this  minimum  value  at  a  level 
such  that  it  will,  in  fact,  be  satisfactory  if  that  is  all  that  is  delivered 
in  the  equipment.  On  the  other  hand,  the  upper  reliability  limit  of  the  effec¬ 
tiveness  renge  should  represent  the  maximum  reliability  value  attainable 
within  the  scope  of  the  plans  and  contractor  proposals  developed  during  program 
definition,  or  as  e  result  of  SPO  program  planning  activities. 

Included  in  Paragraph  3  was  a  brief  indication  that  reliability  is  one 
characteristic  usually  included  as  a  performance  factor;  other  characteristics 
include  speed,  weight,  space,  accurecy,  range,  etc.  This  could  mean,  then, 
that  when  the  performance  factor  incorporates  more  than  one  character is tic* 
the  weight  given  to  any  one  charecterlstic  approaches  an  insignificant  level 
when  considered  within  the  total  Incentive  structure,  and  may  become 
unattractive  for  a  contractor  to  pursue.  Take  the  hypothetical  case  wherein 
the  performance  factor  is  apportioned  25  percent  of  the  total  incentive.  If 
the  performance  factor  is  comprised  of  four  equally  freighted  characteristics 
(one  of  which  is  reliability),  then  any  one  of  them  would  be  afforded 
slightly  more  than  six  percent  of  the  total,  potential  Incan tiva  fae.  It 
would  seem  then,  that  a  contractor  would  find  it  mora  appealing  to  disregard 
those  areas  whsre  small  incentive  fees  could  be  achieved  and,  rather,  pursue 
the  cost  and  schedule  incentive  factors  which  ylald  the  greater  fees.  The 
very  fact  that  the  reliability  characteristic  was  chosan  to  be  incentived 
should  indicate  that  It  was  highly  desireable  to  achiave  greeter  reliability 
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than  is  normally  achieveable  through  the  application  of  standard  techniques. 

In  turn,  an  appropriate  weighting  should  be  afforded  to  it,  not  as  a  part  of 
the  performance  factor,  but  in  terms  of  the  total  multiple  incentive  structure: 
some  contend  that  it  should  be  in  the  order  of  15  to  30  percsnt  of  the  total 
incentive  fee. 

5.  Reliability  Demonstration  and  Incentives  Formulation: 

A  search  of  existing  ESD  contracts  indicates  that  most  existing  incentive 
contracts  have  dealt  with  cost  incentives  alone.  Summarized  briefly  are 
several  interrelated  reasons  for  the  absence  of  reliability  incentives; 

a.  Tbe  difficulty  in  defining  the  reliability  requirements. 

b.  The  principle  of  concurrency  which  does  not  always  permit  mean¬ 
ingful  reliability  demonstration  testings. 

c.  The  absence  of  demonstration  models  for  high  ICBF  situations. 

d.  The  tendency  of  government  negotiations  to  ignore  compliance  to 
reliability  requirements  during  contract  negotiations. 

As  mertioned  earlier,  the  term  "reliability"  is  absolutely  without  con¬ 
tractual  meaning  unless  clearly  defined.  Any  definition  of  rsliability  in 
terms  of  "satisfactory  performance"  is  not  adequate  by  itself.  Reliability  or 
statistical  probability  must  be  defined  in  terms  of  demonstration  conditions 
and  methods,  risks  and  confidences,  and  success  and  failure,  etc.  Therefors, 
it  is  useless  to  begin  a  profit  determination  or  an  incentive  negotiation  for 
reliability  unless  both  contractor  and  ESD  are  first  able  to  agree  on  demonstra¬ 
tion,  and  validity  of  demonstration  or  the  conditions  under  which  collected 
data  will  be  acceptable  or  unacceptable  for  computational  purposes. 

The  quantities  or  values  to  be  selected  for  upper  and  minimum  (lower) 
acceptance  reliability  constitute  the  range  of  "incentive  effectiveness";  ESD 
must  establish  tbese  values.  Once  the  range  of  incentive  effectiveness  has 
been  established,  the  actual  incentive  aspects  are  in  a  position  to  be 
negotiated.  Several  patterns  may  be  established  for  this  formulation.  Figure 
3  displays  fee -reliability  relationship.  Figure  4  displays  a  progressive 
incentive  relationship. 

Both  figures  indicate; 

a.  A  system  of  rewards  and  penalties. 

b.  Establishment  of  an  KTBF  incentive  range  of  effectiveness. 

c.  Establishment  of  a  target  (minimum  acceptance)  KTBF.  Here,  no 
effect  on  target  fee  results  if  this  value  is  demonstrated.  (That  is,  the 
target  fee,  but  no  additional  fee,  is  provided.) 
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Linear  Relationship,  for  Incentive 

With  Reward  and  Penalty 
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Figure  3 


Progressive  Relationship  for  Incentive 

VUfa.  Safari.  Penalty 
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KTBF  (Hours -Normalized) 


Figure  4 
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As  a  numerical  example,  assume  e  target  (minimum  ecceptance)  reliability  of 
90  percent,  a  range  of  effectiveness  of  +  4  percent,  a  target  fee  of  6  percent, 
and  fee  swing  of  +  4  percent.  Based  on  this  information,  Figure  3  is 
restructured  es  follows* 


Lineer  Relationship  for  Incentive 
With  Reward  and  Penalty  -  Specific  Caje 


Figure  5 


Demonstration  of  the  target  (minimum  acceptance) 
on  target  fee,  i.e.,  the  six  percent  fee  is  awarded, 
the  target  velue  produces  e  gradual  reduction  in  fee 
equel  to  two  percent.  The  computation  of  fee  effect 
product 


value  produces  no  effect 
Failure  to  demonstrate 
to  e  velue  of  total  fee 
is  ohtained  from  the 
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where  R  =  Demonstrated  Reliability 

R0  =  Target  (minimum  acceptance)  Value 
Rb  =  Reliability  Ohjective 
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Using  our  previous  values  end  Figure  5,  Tehle  I  can  be  constructed  to 
determine  the  impact  by  various  R  values. 


Effect  on  Fee  bv  Verlous  R  Values 


R  -  Ro  Ro  Effect  Fee  Total  Fee 


a()0 

Ro 

Rh  -  Ro 

on  Fee 

Range  (%) 

P»id  (*) 

86 

-4/90 

90/4 

-vl 

4 

2 

87 

-3/90 

90/4 

-3/4 

4 

3 

88 

-2/90 

90/4 

-1/2 

4 

4 

89 

-1/90 

90/4 

-1/4 

4 

5 

90 

0/90 

90/4 

0 

4 

6 

91 

I/90 

90/4 

+1/4 

4 

7 

92 

2/90 

90/4 

+1/2 

4 

8 

93 

3/90 

90/4 

+3/4 

4 

9 

94 

4/90 

90/4 

+1 

4 
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Underlying  the  reliability  incentive  situation  is  the  question  of  the 
probability  of  demonstrating  reliability  values  within  the  constreints  and 
framework  of  the  demonstration  model.  Since  statistical  sampling  of  at  least 
the  time  domain  is  involved  in  any  demonstration  raoblem,  there  exists  Type 
I  and  Type  II  errors  (producer  end  consumer  risks).  Furthermore ,  to  minimize 
his  risk  of  not  demonstrating  specific  reliability  values  or  maximize  his 
probability  of  receiving  increased  fee  through  incentive  provisions,  e  con¬ 
tractor  will  have  to  consider  designing  for  increased  reliability  beyond  the 
range  of  reliability  interest.  For  example,  essume  a  fixed  test  time  Poisson 
model,  where  total  allowable  test  time  is  set  at  two  multiples  of  desired 
equipment  MTBF  (minimum  acceptance  reliability ).  If  only  one  failure  is 
permitted  during  this  test  time,  end  if  a  contractor  submitted  for  demonstra¬ 
tion  en  equipment  which  had  an  MTBF  equal  to  the  target  (minimum  acceptance) 
MTBF,  he  would  have  only  a  chance  of  obtaining  maximum  fee.  He  would 
have  to  configure  an  equipment  which  had  en  MTBF  of  approximately  twice  the 
value  of  the  target  MTBF  in  order  to  increase  the  probability  to 
Obviously,  this  destroys  the  intent  of  the  incentive. 

The  prohlem,  then,  is  how  to  provide  e  suitable  arrangement  such  that 
various  values  of  MTBF  may  be  demonstrated  with  stipulations  of  incentive  for 
each  of  these  values.  As  en  illustration,  Table  II,  which  is  based  on  the 
Poisson  Sequential  Test  (PST)1,  is  provided* 


5 See  Section  VI,  Reliehility  Deciei on-Making,  Construction  and  Application 
of  Probability  of  Acceptance  Curve*  for  diecueeion  on  PST  criteria. 
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Risk  Level  =  2% 

Total  Observed 
failures 

0 

1 

2 

3 

4 

5 

6 

7 


Discrimination  Ratio  -  3/2 

Total  Test  Time*  Total  Test  Time* 

For  Reject  For  Accept 

(frjuaj-  or  leap)  {Eq^al  or  More) 

2.3 

3.9 


5.3 


2.65 

3.6 


4.8 


5.85 


5.85 


•Total  test  time  is  in  total  hours  of  satisfactory  operating  time  and  is 
expressed  in  multiples  of  minimum  acceptance  reliability  (MTBF). 

Tahle  II 


The  test  would  proceed  according  to  the  specified  Accept -Reject 
Criteria  of  Tshle  II  (or  similar  tahle)  until  such  time  that  an  Accept  or 
Relect  decision  is  reached.  Assuming  that  an  Accept  decision  is  attained, 
the  contractor  may  elect  to  continue  the  test  up  to  a  pre -established  cut-off. 
Whatever  point  he  elects  to  stop  testing,  say  st  Time  t*,  the  number  of 
failures  that  have  occurred  by  that  time,  say  r,  is  used  to  solve  the 
following  equation: 


C(r,*S0Jt»)  = 


s 


e-t,/9°  -  e, 

x! 


The  alternatives  for  incentive  would  be: 

a.  If  c(r;0o;t*)£  5%,  award  maximum  incentive. 

b.  If  c(r;0o;t*)i  10%,  award  intermediate  incentive. 

c.  If  c(r;0Q;t*)  £  15%,  award  minimum  incentive. 


t 


4 
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As  further  illustration,  assume  a  minimum  acceptance  reliability  of  200 
hours  MTBF.  Since  the  discrimination  ratio  of  Tahle  II  is  3/2,  the  upper 
limit  will  be  300  hours  MTBF.  The  range  of  effectiveness,  then,  is  200  to 
300  hours  MTBF.  Assume  that  an  Accept  decision  was  reached,  Table  II  at 
Total  Observed  Failures  =  one  in  780  hours  of  test  time  (3.9  *  200).  In 
order  for  the  contractor  to  obtain  maximum  incentive  for  reliability 
(alternative  a),  he  would  have  to  accomplish  an  additional  645  hours  of 
satisfactory  testing  with  no  additional  failures.  This  was  determined  by  the 
solution  of  the  above  equation  with  r  =  1,  Qo  “  300  and  c(r}3ojt*)  **  5%.  An 
alternate  (and  easier)  method  would  be  to  scan  the  Poisson  tahles  until  a  U 
was  found  with  x  =  1  and  c(x)  =  .05.  It  will  be  found  that  D  =  4.75  at  these 
values.  Then  proceed  as  follows  to  find  t*  (total  test  time): 

'  ’  ‘-7S  ‘  % 
where  0Q  =  300 

therefore,  t*  =  300(4.75) 

=  1425  hours 

But,  3.9(200  hours)  =  780  hours  test  time  has  been  accomplished  with  one 
failure.  Therefore,  the  additional  test  time  while  experiencing  no  addi-  _ 
tional  failures  necessary  to  capture  maximum  incentive  will  be  (1425  ~  780) 

645  hours. 

Table  III  provides  the  Accept -Reject  Criteria  of  Table  II  together  with 
the  additional  railure-free  test  time  at  each  Accept  decision  point  necessary 
to  attain  the  incentive  indicated.  Note  that  no  additional  fee  is  provided 
beyond  the  third  decision  point  (i.e.,  Failures  =  2).  It  should  be  realized 
that  any  number  of  similar  plans  can  be  generated  as  a  result  of  selecting 
different  risk  levels  and  discrimination  ratios  (incentive  range);  what  is 
emphasized  here  is  the  general  technique  that  is  involved. 

The  attractiveness  of  increased  fee  from  reliability  may  be  diminished  by 
the  need  to  expend  additional  funds  and  time  to  insure  a  reasonable  chance 
of  obtaining  maximum  fee  for  reliability.  In  a  multi-incentive  contract 
where  cost  and  schedule  incentives  are  separately  identified  and  given  con- 
sidersble  weight,  while  reliability  is  submerged  as  one  of  several  performance 
characteristics,  a  contractor's  interest  may  lag  in  improving  his  fee  posi¬ 
tion  through  increased  reliability. 

One  method  to  sustain  interest  in  reliability  is  to  make  all  incentive 
fees  for  performance,  schedule,  cost,  etc.,  contingent  upon  the  reliability 
demonstrated.  Mathematically, 

T(total  fee)  =  f^j  (R;  R©;  R^)  f 2  ) 

Where  f2(S;C; — )  is  the  fee  paid  exclusive  of  the  effects  of  reliability. 

If  the  incentives  for  cost,  schedule,  etc.,  are  additive  exclusive  of  the 
effects  of  reliability,  the  total  incentive  fee  is  a  imply: 
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+  Value  of  Reliability  Incentive =  Total  Incentive  Fee 
Where  Fs,  Fc,  etc,  are  the  incentive  fees  for  schedule,  cost,  etc. 

When  the  demonstrated  or  achieved  reliability  is  equal  to  the  relia¬ 
bility  objective,  a  total  incentive  fee  Is  simply  the  maximum  allowed  fee. 
When  the  demonstrated  reliability  is  equal  to  the  target  or  minimum 
ecceptance  reliability,  no  incentive  fee  is  paid  and  total  fee  equals  target 
fee.  Conceivably,  no  incentive  fee  could  result  even  if  demonstrated  relia¬ 
bility  was  in  excess  of  target  reliability.  This  situation  would  arise  if 
a  contractor  failed  to  satisfy  the  requirements  which  permit  the  payment  of 

V  Fc»  etc* 

A  numerical  example  might  illustrate  the  implications  of  the  suggested 
model.  Assume  a  target  fee  of  6%,  a  maximum  fee  of  14%,  schedule  and  cost 
incentives  with  weights  30 %  and  25%,  respectively.  Assume  further  that 
desired  reliability  and  target  reliability  are  98%  and  90%,  respectively, 
with  a  linear  incentive.  If  the  requirements  for  schedule  and  cost  payments 
are  satisfied,  the  total  incentive  and  total  fee  are  as  given  in  Table  IV. 
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Example  of  Total  Fee  Computations  in  *  Multl-Jncer.tlve  Situation 
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HEADQUARTERS 

AIR  FORCE  SYSTEMS  COMMAND 
UNITED  STATES  AIR  FORCE 
ANDREWS  AIR  FORCE  BASE 
Washington  25,  D.C. 


REPLY  TO 
ATTN  OF: 

SCKP 

19  November  1963 

SUBJECT: 

Reliability  Programs  (AFR  80-5) 

TO: 

ASD  SSD  BSD 

(Director  of  Procurement) 

ESD 

1 .  In  a  recent  general  inspection  the  AFSC  Inspector 
General  reported  that: 

"Two  hundred  contracts  were  reviewed  and  only  two 
contained  dollar  penalties  for  failure  to  attain  required 
reliebility  in  either  the  contract  or  the  statement  of 
work  thet  pertained  to  reliebility.  Few  contracts  con¬ 
tained  specific  Mean  Time  Between  Failure  (KTBF)  figures 
or  e  clearly  stated  mathematical  model  for  reliability 
computation. " 

2.  Your  attention  is  invited  to  AFR  80-5  and  in  particular 
paregrephs  4f,  6c  end  e.  To  assure  that  this  Command  ful¬ 
fills  its  responsibilities  under  this  program,  it  is 
requested  that  every  reasonable  effort  be  made  to  negotiate 
the  appropriete  fee tors  into  new  contracts  and,  where  not 
now  in  evidence,  to  negotiate/Venegotiate  adequate  dollar 
penalties  and  administrative  requirements  to  satisfy  the 
celculations  of  Mean  Time  Between  Failure  factors. 

FOR  THE  COMMANDER 

/s/  Herbert  L.  Repet ti 

HERBERT  1.  RE PETTI 

Deputy  Director  of  Procurement 

DCS/Procurement  &  Production 
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Excerpts  From 

ARMED  SERVICES  PROCUREMENT  REGULATION 
Section  VII,  Paragraph  7-103.5 

Paragraph  5.  INSPECTION 

(a)  All  supplies  (which  term  throughout  this  clause  includes  without 
limitation  raw  materials,  components,  intermediate  assemblies,  and  end 
products)  shall  be  subject  to  inspection  and  test  by  the  Government,  to  the 
extent  practicable  at  all  times  and  places  including  the  period  of  manufacture, 
and  in  any  event  prior  to  acceptance. 

(h)  In  case  any  supplies  or  lots  of  supplies  are  defective  in  material 
or  workmanship  or  otherwise  not  in  conformity  with  the  requirements  of  this 
contract,  the  Government  shall  have  the  right  either  to  reject  them  (with  or 
without  instructions  as  to  their  disposition)  or  to  require  their  correction. 
Supplies  or  lots  of  supplies  which  have  been  rejected  or  required  to  be 
corrected  shall  be  removed  or,  if  permitted  or  required  by  the  Contracting 
Officer,  corrected  in  place  by  and  at  the  expense  of  the  Contractor  promptly 
after  notice,  and  shall  not  thereafter  be  tendered  for  acceptance  unless  the 
former  rejection  or  requirement  of  correction  is  disclosed.  If  the  Contractor 
fails  promptly  to  remove  such  supplies  or  lots  of  supplies  which  are  required 
to  be  removed,  or  promptly  to  replace  or  correct  such  supplies  or  lots  of 
supplies,  the  Government  either  (i)  may  hy  contract  or  otherwise  replsce  or 
correct  such  supplies  and  charge  to  the  Contractor  the  cost  occasioned  the 
Government  thereby,  or  (ii)  may  terminate  this  contract  for  default  as  pro¬ 
vided  in  the  clause  of  this  contract  entitled  "Default."  Unless  the  Con¬ 
tractor  corrects  or  replaces  such  supplies  within  the  delivery  schedule,  the 
Contracting  Officer  may  require  the  delivery  of  such  supplies  at  a  reduction 
in  price  which  is  equitable  under  the  circumstances.  Failure  to  agree  to  such 
reduction  of  price  shall  be  a  dispute  concerning  a  question  of  fact  within  the 
meening  of  the  clause  of  this  contract  entitled  "Disputes." 

(c)  If  any  inspection  or  test  is  made  by  the  Government  on  the  premises 
of  the  Contractor  or  a  subcontractor,  the  Contractor  without  additional  charge 
shall  provide  all  reasonable  facilities  and  assistance  for  the  safety  and 
convenience  of  the  Government  inspectors  in  the  performance  of  their  duties. 

If  Government  inspection  or  test  is  made  at  a  point  other  than  the  premises 
of  the  Contractor  or  a  subcontractor,  it  shall  be  at  the  expense  of  the 
Government  except  as  otherwise  provided  in  this  contracts  Provided.  That  in 
case  of  rejection  the  Government  shall  not  be  liable  for  any  reduction  in 
value  of  samples  used  in  connection  with  such  inspection  or  test.  All  inspec¬ 
tions  and  tests  hy  the  Government  shall  be  performed  in  such  a  manner  as  not 
to  unduly  delay  the  work.  The  Government  reserves  the  right  to  charge  to  the 
Contractor  any  additional  cost  of  Government  inspection  and  test  when  supplies 
ere  not  ready  at  the  time  such  inspection  and  test  is  requested  by  the  Con¬ 
tractor  or  when  reinspection  or  retest  is  necessitated  by  prior  rejection. 
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Acceptance  or  rejection  of  the  supplies  shall  be  made  as  promptly  as 
practicable  after  delivery,  except  as  otherwise  provided  in  this  contract; 
but  failure  to  inspect  and  accept  or  reject  supplies  shall  neither  relieve 
tha  Contractor  from  responsibility  for  such  supplies  as  are  not  in  accordance 
with  tha  contract  requirements  nor  impose  liability  on  the  Government  therefor. 

(d)  The  inspection  and  test  by  the  Government  of  any  supplies  or  lots 
thereof  does  not  relieve  the  Contractor  from  any  responsibility  regarding 
defects  or  other  failures  to  meet  the  contract  requirements  which  may  be  dis¬ 
covered  prior  to  acceptance.  Except  as  otherwise  provided  in  this  contract, 
acceptance  shall  be  conclusive  except  as  regards  latent  defects,  fraud,  or  such 
gross  mistakes  as  amount  to  fraud. 

(e)  The  Contractor  shall  provide  and  maintain  an  inspection  system 
acceptable  to  the  Government  covering  the  supplies  hereunder.  Records  of  all 
inspection  work  by  the  Contractor  shall  be  kept  complete  and  available  to  the 
Government  during  the  performance  of  this  contract  and  for  such  longer  period 
as  may  be  specified  elsewhere  in  this  contract. 

Paragraph  1 1 .  DEFAULT 

(e)  The  Government  may,  subject  to  the  provisions  of  paragraph  (c)  below, 
by  written  notice  of  default  to  the  Contractor,  terminate  the  whole  or  any 
part  of  this  contrect  in  any  one  of  the  following  circumstances: 

(i)  if  the  Contractor  fails  to  make  delivery  of  the  supplies  or  to 
perform  the  services  within  the  time  specified  herein  or  any  extension 
thereof;  or 

(ii)  if  the  Contractor  fails  to  perform  any  of  the  other  provisions 
of  this  contract,  or  so  fails  to  make  progress  as  to  endanger  performance 
of  this  contract  in  accordance  with  its  terms,  and  in  either  of  these 
two  circumstances  does  not  cure  such  failure  within  a  period  of  10  days 
(or  such  longer  period  as  the  Contracting  Officer  may  authorize  in  writing) 
after  receipt  of  notice  from  the  Contracting  Officer  specifying  such 
feilure. 

(b)  In  the  event  the  Government  terminates  this  contract  in  whole  or  in 
part  as  provided  in  paragraph  (a)  of  this  clause,  the  Government  may  procure, 
upon  such  terms  and  in  such  manner  as  the  Contracting  Officer  may  deem 
appropriate,  supplies  or  services  similar  to  those  so  terminated,  and  the  Con¬ 
tractor  shall  be  liable  to  the  Government  for  any  excess  costs  for  such  similar 
supplies  or  services:  Provided .  That  tha  Contractor  shall  continue  the 
performance  of  this  contract  to  the  extent  not  terminated  under  the  provisions 
of  this  clause. 

(c)  Except  with  respect  to  defaults  of  subcontractors,  the  Contractor 
shall  not  be  liable  for  any  excess  coats  if  the  failure  to  perform  the  contract 
arises  out  of  causes  beyond  the  control  and  without  the  fault  or  negligence 

of  the  Contractor.  Such  causes  may  include,  but  ere  not  restricted  to,  acts  of 
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God  or  of  the  public  enemy,  acts  of  the  Government  in  either  its  sovereign 
or  contrectual  capacity,  fires,  floods,  epidemics,  quarentine  restrictions, 
strikes,  freight  embargoes,  and  unusually  severe  weather;  but  in  every  case 
the  feilure  to  perform  must  be  beyond  the  control  end  without  feult  or 
negligence  of  the  Contrector.  If  tha  feilure  to  perform  is  ceused  by  the 
defeult  of  a  subcontractor,  and  if  such  defeult  arises  out  of  causes  beyond  the 
control  of  both  the  Contractor  and  subcontractor,  end  without  the  fault  or 
negligence  of  either  of  them,  the  Contractor  shell  not  be  lieble  for  any  excess 
costs  for  feilure  to  perform,  unless  the  supplies  or  services  to  be  furnished 
by  the  subcontractor  were  obtainable  from  other  sources  in  sufficient  time  to 
permit  the  Contrector  to  meet  the  required  delivery  schedule. 

(d)  If  this  contrect  is  terminated  as  provided  in  pare graph  (e)  of  this 
cleuse,  the  Government,  in  eddition  to  any  other  rights  provided  in  this  cleuse, 
may  require  the  Contrector  to  trensfer  title  end  deliver  to  the  Government, 

in  the  manner  end  to  the  extent  directed  by  the  Contracting  Officer,  (i)  any 
completed  supplies,  end  (ii)  such  partielly  completed  supplies  end  materials, 
parts,  tools,  dies,  jigs,  fixtures,  plens ,  drawings,  information,  end  contract 
rights  (hereinafter  called  "manufacturing  materials ")  es  the  Contrector  has 
specifically  produced  or  specifically  acquired  for  the  performance  of  such 
part  of  this  contract  as  hes  been  terminated;  end  the  Contractor  shall,  upon 
direction  of  the  Contracting  Officer,  protect  and  preserve  property  in  posses¬ 
sion  of  the  Contractor  in  which  the  Government  has  en  interest.  Payment  for 
completed  supplies  delivered  to  and  accepted  by  the  Government  shell  be  et 
the  contract  price.  Payment  for  manufacturing  materiels  delivered  to  and 
accepted  by  the  Government  end  for  the  protection  end  preservation  of  property 
shall  ba  in  en  amount  egreed  upon  by  the  Contrector  and  Contrecting  Officer; 
feilure  to  spree  to  such  amount  shall  be  e  dispute  concerning  a  question  of 
fact  within  the  meaning  of  the  cleuse  of  this  contract  entitled  "Disputes." 

(e)  If,  efter  notice  of  termination  of  this  contrect  under  the  provisions 
of  paregraph  (e)  of  this  cleuse,  it  is  determined  that  tha  failure  to  perform 
this  contract  is  due  to  causes  beyond  the  control  end  without  the  feult  or 
negligence  of  the  Contrector  or  subcontrector  pursuant  to  tha  provisions  of 
paregraph  (c)  of  this  cleuse,  such  notice  of  default  shell  be  deemed  to  have 
been  issued  pursuent  to  the  cleuse  of  this-  contract  entitled.  "Termination  for 
Convenience  of  the  Government,"  end  the  rights  and  obligations  of  the  parties 
hereto  shall  in  such  event  be  governed  by  such  cleuse.  (Except  es  otherwise 
provided  in  this  contract,  this  peregreph  (e)  applies  only  if  this  contrect 

contains  such  cleuse.) 

(f)  The  rights  end  remedies  of  the  Government  provided  in  this  cleuse 
shell  not  be  exclusive  end  ere  in  addition  to  eny  other  rights  end  remedies 
provided  by  law  or  under  this  contract. 

Peregreph  12.  DISPUTES 

(e)  Except  es  otherwise  provided  in  this  contract,  eny  dispute  concerning 
e  question  of  feet  erising  under  this  contrect  which  is  not  disposed  of  by 
agreement  shall  be  decided  by  tha  Contracting  Officer,  who  shall  reduce  his 
decision  to  writing  end  mail  or  otherwise  furnish  e  copy  thereof  to  tha  Con¬ 
trector.  The  decision  of  the  Contracting  Officer  shall  be  final  and  conclusive 
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unless,  within  30  days  from  the  date  of  receipt  of  such  copy,  the  Contractor 
mails  or  otherwise  furnishes  to  the  Contracting  Officer  a  written  appeal 
addressed  to  the  Secretary.  The  decision  of  the  Secretary  or  his  duly 
authorized  representative  for  the  determination  of  such  appeals  shall  be 
final  and  conclusive  unless  determined  by  a  court  of  competent  jurisdiction 
to  have  been  fraudulent,  or  capricious,  or  arbitrary,  or  so  grossly 
erroneous  as  necessarily  to  imply  bad  faith,  or  not  supported  by  substantial 
evidence.  In  connection  with  any  appeal  proceeding  under  this  clause,  the 
Contrsctor  shall  be  afforded  an  opportunity  to  be  heard  and  to  offer  evidence 
in  support  of  its  appeal.  Pending  final  decision  of  e  dispute  hereunder,  the 
Contractor  shall  proceed  diligently  with  the  performance  of  the  contract  and 
in  accordance  with  the  Contracting  Officer's  decision. 

(b)  This  "Disputes"  clause  does  not  preclude  consideration  of  lew 
questions  in  connection  with  decisions  provided  for  in  paragraph  (a)  above: 
Provided .  That  nothing  in  this  contract  shall  be  construed  as  making  final 
the  decision  of  any  administrative  official,  representative,  or  board  on  a 
question  of  law. 
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